background image

Content tagged with: import

Eric's picture

In anticipation of my new iPhone 4 (and a larger hard drive), I decided to write a script to help choose which albums I should include. Through iTunes you can export your library as text, which can then be imported into MySQL, and SQL can be executed to show.. well, whatever your heart desires!

I started by exporting my entire music library, and chose Plain Text as the format. This iTunes functionality creates a tab separated list of columns and data.

I then created a new MySQL database and table (SEE: TABLE_NAME) to contain my iTunes song data. I also added an additional column as a primary key, called sql_id.

Next, I created a PHP script to parse the text file, dynamically add the columns to my table, and import all my song data. This code uses PEAR's DB library for the database access layer.

<?php
// define table name
define('TABLE_NAME', 'itunes');

// create database connection
require_once('DB.php');
$dsn = 'mysqli://db_user:db_password@db_host/database_name';
$DB =& DB::connect($dsn);
if (
DB::isError($DB)) {
  die(
$DB->getMessage());
}
$DB->setFetchMode(DB_FETCHMODE_ASSOC);

// load text file
$file = file_get_contents('Music.txt');

// explode on new line
$file = explode("\r", $file);

// get a list of existing columns
$sql = "show columns in " . TABLE_NAME;
$result = $DB->getAll($sql);
$existing_columns = array();
foreach (
$result as $key => $value) {
 
$existing_columns[] = $value['Field'];
}

// create a variable to contain sql column names
$sql_column_name = array();

// loop through each line in the file
foreach ($file as $key => $value) {

 
// explode on tab to get column list
 
$exploded = explode("\t", $value);

 
// check for first row, which contains column headers
 
if ($key == 0) {

   
// loop through column list and ensure they exist
   
foreach ($exploded as $new_table) {
   
     
// create a new column name without spaces
     
$new_table = str_replace(' ' , '_', $new_table);

     
// check if the new column should be added   
     
if (!in_array($new_table, $existing_columns)) {
          
       
// define sql to add new column
       
$sql = "alter table " . TABLE_NAME . " add column `$new_table` varchar(255) default null";
       
$result = $DB->query($sql);
       
       
// check for error
       
if (!$result) {
          echo
"<pre>" . print_r($result, true) . "</pre>";
          die;
        }
               
       
// define sql to add index
       
$sql = "alter table " . TABLE_NAME . " add index `index_$new_table` ($new_table)";
       
$result = $DB->query($sql);
       
       
// check for error
       
if (!$result) {
          echo
"<pre>" . print_r($result, true) . "</pre>";
          die;         
        }
     
      }
// end if
     
     
$sql_column_names[] = $new_table;
   
    }
// end foreach
     
 
} // end if ($key[0])
 
else {
   
   
// prepare values to insert
   
$insert_values = array();
    foreach (
$exploded as $k => $v) {
     
$insert_values[$k] = $DB->quoteSmart($v);
    }
   
   
// define SQL to insert data into table
   
$sql = "insert into " . TABLE_NAME . " (" . implode(',', $sql_column_names) . ") values (" . implode(',', $insert_values) . ")";
   
   
// execute sql
   
$result = $DB->query($sql);
   
   
// check for SQL error
   
if (!$result) {
      echo
"<pre>" . print_r($result, true) . "</pre>";
    }
 
  }

}
?>

Now, all my iTunes data is accessible by SQL, sweet. I decided to create a view showing my albums with an average ranking.

create view album_rankings as
select Artist, Album, Genre, avg(My_Rating) as album_rating, count(*) as countX
from itunes
where Album != 'misc'
group by Artist, Album
order by avg(My_Rating) desc, countX desc

And finally, I executed the following SQL to show my highest rated albums.

select *
from album_rankings
where countX > 3

Top Rated Albums

Eric's picture

In this code snippet, I'll show how you can parse a (large) CSV file using Drupal's Batch API. The purpose of batching an operation is to avoid PHP memory limits and time outs. Before you begin, I recommend reviewing the following two articles. Be sure to review the additional batch parameters outlined in the documentation, you might need to use them.

Batch API
Batch Operations

We can start by defining the arguments that will be passed into the batch_set() function. For this example, I added this code in an arbitrary page callback function.

<?php
function MYMODULE_callback_csv_import() {

 
// define path to CSV file
 
$csv_file_path = file_directory_path() . '/import_path/myfile.csv';

 
// define a redirect path upon batch completion
 
$redirect_path = 'admin/import-csv';

 
// define batch array structure
  // NOTE: minimal parameters defined to simplify code
 
$batch = array(
   
'title' => t('Reading File'),
   
'operations' => array(
      array(
       
'_MYMODULE_batch_read', array($csv_file_path),
      ),
    ),
  );

 
// set batch
 
batch_set($batch);

 
// process batch
 
batch_process($redirect_path);

}
?>

Next, we'll define the batch callback function. This function will be called repeatedly until the $context['finished'] variable is set to "1".

<?php
function _MYMODULE_batch_read($csv_file_path, &$context) {
 
 
// define batch limit
 
$batch_limit = 100;

 
// assume the batch process has not completed
 
$context['finished'] = 0;

 
// open the file for reading
 
$file_handle = fopen($csv_file_path, 'r');

 
// check if file pointer position exists in the sandbox, and jump to location in file
 
if ($context['sandbox']['file_pointer_position']) {
   
fseek($file_handle, $context['sandbox']['file_pointer_position']);
  }
 
 
// loop through the file and stop at batch limit
 
for ($i = 0; $i < $batch_limit; $i++) {

   
// get file line as csv
   
$csv_line = fgetcsv($file_handle);

   
// NOTE: at this point, do what ever you'd like with the CSV array data!
   
if (is_array($csv_line)) {
     
// db_query(), etc   
   
}

   
// retain current file pointer position
   
$context['sandbox']['file_pointer_position'] = ftell($file_handle);

   
// check for EOF
   
if (feof($file_handle)) {
     
// complete the batch process
     
$context['finished'] = 1;

     
// end loop
     
break;
    }

  }

}
?>

The batch operation will be called until the end of the CSV file is reached. The $context variable is passed by reference into the batch callback so you can maintain data through each iteration; in this case, the position of the file pointer. When the batch operation is complete, the user will be redirected to the batch_process() path argument.

It's important to read the full Batch API documentation so you can take advantage of its additional features: finished callback, init_message, progress_message, error_message, etc.