​

Table Upload Methods: Two Approaches

​

Method 1: CSV Upload - Simple but Inflexible

​

How It Works

• UI Upload: Click “Upload Data” in Dune interface, select CSV file, name your table, add description, and save
• API Upload: Use the /upload endpoint to programmatically upload CSV files
• Schema: Automatically inferred from CSV data (no manual schema definition required)
• Table Naming: Results in dune.team_name.dataset_name format

​

Key Characteristics

• Simplicity: Minimal setup required - just upload and go
• User-Friendly: Perfect for non-technical users
• Quick Start: Immediate data availability for querying
• Automatic Processing: Schema inference handles data types automatically
• Both UI and API: Available through web interface or programmatic upload

• Size Constraint: Maximum 200MB file size limit
• No Incremental Updates: Must replace entire table to update data
• Limited Control: Cannot modify schema after upload
• File Format: Only supports CSV format
• Inflexible Schema: Stuck with automatically inferred data types
• No Append Operations: Cannot add new rows to existing data

​

Best Use Cases

• Small to medium datasets (under 200MB)
• One-time data uploads
• Static datasets that don’t need frequent updates
• Prototyping and experimentation
• Users without programming experience

​

Credit Usage

• Plus Plan: 5,000 data points per credit
• Premium Plan: 25,000 data points per credit

​

Method 2: Programmatic Table Management - Flexible but Complex

​

Core Operations

​

1. Create Table (POST /v1/table/create)

• Define table schema explicitly with column names, types, and nullability
• Set table as public or private
• Specify namespace and table name
• Cost: 10 credits per successful table creation

​

2. Insert Data (POST /v1/table/insert)

• Add new rows to existing tables
• Support for incremental data loading
• Append operations without replacing existing data
• JSON payload with structured data

​

3. Clear Data (POST /v1/table/clear)

• Remove all data from table while preserving schema
• Useful for complete data refreshes
• Table structure remains intact

​

4. Delete Table (DELETE /v1/table/delete)

• Permanently remove table and all its data
• Complete cleanup operation
• Irreversible action

​

Key Characteristics

• Full Control: Complete management of table lifecycle
• Incremental Updates: Add data without replacing existing content
• Large Data Support: No 200MB size limitation
• Custom Schema: Define exact data types, constraints, and structure
• Append Operations: Add new rows while keeping existing data
• Flexible Updates: Clear and refresh data as needed
• Programmatic: Perfect for automated workflows and ETL pipelines
• Privacy Control: Set tables as public or private

• Complexity: Requires more technical knowledge and setup
• Schema Management: Must manually define and maintain schema
• Multiple API Calls: Requires orchestration of multiple endpoints
• Higher Learning Curve: More complex implementation
• Credit Consumption: Table creation costs 10 credits

​

Typical Workflow

1. Create table with defined schema
2. Insert initial data batch
3. Insert additional data as needed (append operations)
4. Clear table when full refresh is needed
5. Insert new complete dataset
6. Delete table when no longer needed

​

Best Use Cases

• Large datasets (over 200MB)
• Frequently updated data requiring incremental loads
• Production ETL pipelines
• Applications needing precise schema control
• Automated data workflows
• Complex data structures with specific type requirements

​

Comparison Matrix

​

Decision Framework

​

Choose CSV Upload When:

• Dataset is under 200MB
• Data updates are infrequent
• Quick prototyping or one-off analysis
• Non-technical user or minimal programming resources
• Simple data structure with standard types
• Time to market is critical

​

Choose Programmatic Management When:

• Dataset exceeds 200MB
• Regular data updates required
• Part of automated data pipeline
• Need precise control over data types
• Building production applications
• Require incremental data loading
• Complex data workflows with dependencies

​

Implementation Considerations

• CSV Upload: Limited by file size and processing time
• Programmatic: Scalable for large datasets, can handle concurrent operations

• CSV Upload: Minimal monitoring needed
• Programmatic: Requires error handling, retry logic, and workflow orchestration

• Both methods consume credits based on data volume
• Programmatic method has additional table creation costs
• Consider credit consumption in high-frequency update scenarios

​

Recommendation