I am using DolphinDB to aggregate 5-minute K-line data into weekly bars. My DFS table dfs://um (table name k) is partitioned by Month.

I wrote a MapReduce script to perform the aggregation and append the results back to the same table with a different frequency label (freq="w"). However, I noticed that for weeks that span across two months, duplicate records are generated.

For example, the week starting on 2020.01.27 (Monday) spans from late January to early February. In the output, there are two separate rows for 2020.01.27.

The problematic code:

klines = loadTable("dfs://um", "k")
ds = sqlDS()

def calcOHLC1w(t){
        // Aggregating 5-min data to 1-week
        tmp = select "w" as freq, first(open) as open, max(high) as high, min(low) as low, last(close) as close, sum(quote_volume) as quote_vol, sum(count) as count, sum(taker_buy_quote_volume) as taker_buy_vol 
              from t group by weekBegin(open_time) as open_time, ticker
        
        // Appending inside the Map function
        loadTable("dfs://um", "k").append!(tmp)
        return tmp.size()
}

def finalFunc(size){
        print(size)
}

mr(ds, calcOHLC1w, +, finalFunc, false)

Observations:

Duplicates Found: When querying the result, weeks like 2020.01.27 and 2020.02.24 appear twice.
error result 

Correct Aggregation via SQL: If I run a standard distributed SQL query without MapReduce, the result is correct (no duplicates):
// This works correctly
t = select first(open) from klines 
    where freq="5" and open_time <= timestamp(2020.04.01) 
    group by weekBegin(open_time), ticker



correct result
My Questions:

Why does the MapReduce approach generate duplicates while the standard SQL group by does not?


              
      

          
                  time-series
                mapreduce
                dolphindb
                
        
      
    
    
    
    

    

      
Your Answer
      

        

          
          Write your answer...
        
        
          Submit Answer
        
      
    
    


    

      lock_outline
      
You need to log in to submit an answer.
      
        Log In
      
    
    
    
    
    
    

        





        
        
        
        
        


 
  

  
  








  


        
                
      
      
            
            Subscribe