cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
dcrezee
by • New Contributor III
  • 6658 Views
  • 11 replies
  • 3 kudos

Resolved! Issue with quotes in struct type columns when using ODBC

I'm trying to connect to Databricks using pyodbc and I'm running into an issue with struct columns. As far as I understand, struct columns and array columns are not supported by pyodbc, but they are converted to JSON. However, when there are nested c...

Data Engineering
Reply
  • 6658 Views
  • 11 replies
  • 3 kudos
View Replies
Latest Reply
BilalAslamDbrx
Honored Contributor III
  • 3 kudos

@Derk Crezee​ - I learned something today. Apparently ODBC does not convert to JSON. There is no defined spec on how to return complex types, in fact that was added only in SQL 2016. That's exactly what you are running into!End of history lesson Her...

  • 3 kudos
10 More Replies
RicksDB
by • Contributor II
  • 3823 Views
  • 9 replies
  • 1 kudos

Configure jobs throttling for ephemeral cluster ETLs

Hi,Is it possible to configure job throttling in order to queue jobs across a workspace after a given number of concurrent execution when using the ephemeral cluster pattern? The reason is mainly for cost control. We prefer reducing performance rathe...

Data Engineering
Reply
  • 3823 Views
  • 9 replies
  • 1 kudos
View Replies
Latest Reply
RicksDB
Contributor II
  • 1 kudos

Thanks for the help josephk. I will continue to use an interactive cluster for the time being until the release of that new feature. Hopefully, it will allow my use case. Is there visibility on the roadmap for an ETA or more information on it?

  • 1 kudos
8 More Replies
barashe
by • New Contributor II
  • 1168 Views
  • 1 replies
  • 0 kudos

Installing python modules on databricks job clusters

Different than all-purpose clusters, the databricks job new cluster configuration window does not have a "Libraries" tab, in which specific python modules could be installed. What's the best practice for installing python modules on such clusters?

Data Engineering
Reply
  • 1168 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
barashe
New Contributor II
  • 0 kudos

It turns out that the option exists outside of the cluster configuration scope, in the task configuration window itself - under "Advanced options" -> "Add dependent libraries".

  • 0 kudos
guruv
by • New Contributor III
  • 16873 Views
  • 4 replies
  • 5 kudos

Resolved! parquet file to include partitioned column in file

HI,I have a daily scheduled job which processes the data and write as parquet file in a specific folder structure like root_folder/{CountryCode}/parquetfiles. Where each day job will write new data for countrycode under the folder for countrycodeI am...

Data Engineering
Reply
  • 16873 Views
  • 4 replies
  • 5 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 5 kudos

Most external consumers will read partition as column when are properly configured (for example Azure Data Factory or Power BI).Only way around is that you will duplicate column with other name (you can not have the same name as it will generate conf...

  • 5 kudos
3 More Replies
RantoB
by • Valued Contributor
  • 7996 Views
  • 19 replies
  • 7 kudos

Resolved! unzip twice the same file not executing

Hi, I need to unzip some files that are ingested but when I unzip twice the same zipped file, the unzip command does not execute :As suggesgted in the documentation I did :import urllib urllib.request.urlretrieve("https://resources.lendingclub.com/L...

Data Engineering
Reply
  • 7996 Views
  • 19 replies
  • 7 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 7 kudos

Hi @Bertrand BURCKER​ , Create a script.sh and copy the script in the directory where is data.zip archive. This script is working with any name of archives and any name of csv.#!/bin/bash   currLoc="$PWD" path="${currLoc}"   cd ${currLoc}   #EXTRACT ...

  • 7 kudos
18 More Replies
marsjuli
by • New Contributor II
  • 17411 Views
  • 5 replies
  • 3 kudos

Resolved! How to handle <IPython.core.display.HTML object>

Some libraries have intermediate IPython HTML-objects returned to the notebook cell output.Since this happens during training a machine learning model the statements are typically buried within in the library so I cannot easily interfere. (e.g. in or...

grafik.png
Data Engineering
Reply
  • 17411 Views
  • 5 replies
  • 3 kudos
View Replies
Latest Reply
marsjuli
New Contributor II
  • 3 kudos

Hi @Kaniz Fatma​ ,thanks for showing me the link. This helps if you are in control of the generated html-object. If the html-content comes from a library, that is where the problems start, because I cannot wrap displayHTML().(I can of course look for...

  • 3 kudos
4 More Replies
Development
by • New Contributor III
  • 608 Views
  • 0 replies
  • 0 kudos

Hi All, I hope you&#39;re doing well I am facing issue while installing an python library on ADB Cluster. lib - PyCaret ( latest version) its not gett...

Hi All,I hope you're doing wellI am facing issue while installing an python library on ADB Cluster.lib - PyCaret ( latest version)its not getting install and showing me 'Failed' Status.It would be great if you can help here !!Thanks

Data Engineering
Reply
  • 608 Views
  • 0 replies
  • 0 kudos
TimK
by • New Contributor II
  • 3322 Views
  • 3 replies
  • 1 kudos

Resolved! Cannot Get Databricks SQL to read external Hive Metastore

I have followed the documentation and using the same metastore config that is working in the Data Engineering context. When attempting to view the Databases, I get the error:Encountered an internal errorThe following information failed to load:The li...

Data Engineering
Reply
  • 3322 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
TimK
New Contributor II
  • 1 kudos

@Bilal Aslam​  I didn't think to look there before since I hadn't tried to run any queries. I see the failed SHOW DATABASES queries in history and they identify the error: Builtin jars can only be used when hive execution version == hive metastore v...

  • 1 kudos
2 More Replies
KaushalPatidar
by • New Contributor II
  • 2793 Views
  • 7 replies
  • 0 kudos

I cannot access my account, please help

Hi, when I am trying to log into my account, its showing "invalid email address and password". But,I am sure everything is correct. I request @Kaniz Fatma​ , @Harikrishnan Kunhumveettil​ and @Prabakar Ammeappin​ to please look into it and resolve thi...

Data Engineering
Reply
  • 2793 Views
  • 7 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Kaushal Patidar​ , Did you try resetting your password using this Forgot Password link?

  • 0 kudos
6 More Replies
grandsurgical
by • New Contributor
  • 401 Views
  • 0 replies
  • 0 kudos

Grand Surgical, established in 2010, has been manufacturing high-quality Surgical instruments for all disciplines of surgery. Cardiac, Vascular, denta...

Grand Surgical, established in 2010, has been manufacturing high-quality Surgical instruments for all disciplines of surgery. Cardiac, Vascular, dental, ophthalmic.We develop and deliver hospitals and medical professionals worldwide with superior qua...

Data Engineering
Reply
  • 401 Views
  • 0 replies
  • 0 kudos
theclubprice
by • New Contributor
  • 442 Views
  • 0 replies
  • 0 kudos

The Club Price is a leading supplier of high-quality, affordable products whose clientele spans all over Texas and neighboring areas since 1992. We ha...

The Club Price is a leading supplier of high-quality, affordable products whose clientele spans all over Texas and neighboring areas since 1992. We have an exemplary track record of increasing our partners’ sales with our variety in products. We have...

Data Engineering
Reply
  • 442 Views
  • 0 replies
  • 0 kudos
rednirusmart
by • New Contributor
  • 467 Views
  • 0 replies
  • 0 kudos

Rednirus Mart is a Third-Party Pharma Manufacturer and Supplier. If you are looking For Pharma Contract manufacturers For Ayurvedic Medicine Manufactu...

Rednirus Mart is a Third-Party Pharma Manufacturer and Supplier. If you are looking For Pharma Contract manufacturers For Ayurvedic Medicine Manufacturer Company in your region. Rednirus Mart is one of the leading one and their products are manufactu...

Third Party Pharma Manufacturers
Data Engineering
Reply
  • 467 Views
  • 0 replies
  • 0 kudos
pjp94
by • Contributor
  • 2126 Views
  • 1 replies
  • 3 kudos

Use '%sql' inside a python cmd cell?

Hi so I want to essentially execute a sql query if a condition is met. So one of my cells in my python notebook is a sql query (%sql followed by the query). Is there any way to put that in an 'IF' statement ie if an environment variable = some value,...

Data Engineering
Reply
  • 2126 Views
  • 1 replies
  • 3 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 3 kudos

in python cell just use:query = "SELECT 1"spark.sql(query)

  • 3 kudos
RasmusOlesen
by • New Contributor III
  • 6989 Views
  • 4 replies
  • 2 kudos

Upgrading from Spark 2.4 to 3.2: Recursive view errors when using

We get errors like this,Recursive view `x` detected (cycle: `x` -> `x`).. in our long-term working code, that has worked just fine in Spark 2.4.5 (Runtime 6.4), when we run it on a Spark 3.2 cluster (Runtime 10.0).It happens whenever we have,<x is a ...

Data Engineering
Reply
  • 6989 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
arkrish
New Contributor II
  • 2 kudos

This is a breaking change introduced in Spark 3.1 From Migration Guide: SQL, Datasets and DataFrame - Spark 3.1.1 Documentation (apache.org)In Spark 3.1, the temporary view will have same behaviors with the permanent view, i.e. capture and store runt...

  • 2 kudos
3 More Replies
Ryan_Chynoweth
by • Esteemed Contributor
  • 1186 Views
  • 1 replies
  • 0 kudos

Azure_DAAM

Attached to this post we have added an ADLS Gen2 access recommendation to have the ideal security and governance over your data. The best practice involves leveraging Cluster ACLs, cluster configuration, and secret ACLs to handle user access over you...

Data Engineering
Reply
  • 1186 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Ryan Chynoweth​ , Thank you for posting this!

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels