cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

maranBH
by New Contributor III
  • 3263 Views
  • 3 replies
  • 1 kudos

Resolved! Trained model artifact, CI/CD and Databricks without MLFlow.

Hi all,We are constructing our CI/CD pipelines with the Repos feature following this guide:https://databricks.com/blog/2021/09/20/part-1-implementing-ci-cd-on-databricks-using-databricks-notebooks-and-azure-devops.htmlI'm trying to implement my pipes...

  • 3263 Views
  • 3 replies
  • 1 kudos
Latest Reply
sean_owen
Databricks Employee
  • 1 kudos

So you are managing your models with MLflow, and want to include them in a git repository?You can do that in a CI/CD process; it would run the mlflow CLI to copy the model you want (e.g. model:/my_model/production) to a git checkout and then commit i...

  • 1 kudos
2 More Replies
dimsh
by Contributor
  • 14085 Views
  • 3 replies
  • 1 kudos

Resolved! Delta Table is not available in the Databricks SQL

Hi, there!I'm trying to read a data (simple SELECT * FROM schema.tabl_a) from the "Queries" Tab inside the Databricks SQL platform, but always getting "org.apache.spark.sql.AnalysisException: dbfs:/.../.. doesn't exist" DescribeRelation true, [col_na...

  • 14085 Views
  • 3 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Because it's a delta table, you don't need to provide the schema.

  • 1 kudos
2 More Replies
RicksDB
by Contributor III
  • 7194 Views
  • 6 replies
  • 6 kudos

Resolved! SingleNode all-purpose cluster for small ETLs

Hi,I have many "small" jobs than needs to be executed quickly and at a predictable low cost from several Azure Data Factory pipelines. For this reason, I configured a small single node cluster to execute those processes. For the moment, everything se...

image
  • 7194 Views
  • 6 replies
  • 6 kudos
Latest Reply
RicksDB
Contributor III
  • 6 kudos

@Bilal Aslam​  In my case, it usually depends on the customers and their SLA. Most of them usually do not have a "true" high SLA requirement thus prefer the jobs to be throttled when the actual cost is within a certain range of the budget instead of ...

  • 6 kudos
5 More Replies
Anonymous
by Not applicable
  • 9965 Views
  • 7 replies
  • 3 kudos

Resolved! Issue with quotes in struct type columns when using ODBC

I'm trying to connect to Databricks using pyodbc and I'm running into an issue with struct columns. As far as I understand, struct columns and array columns are not supported by pyodbc, but they are converted to JSON. However, when there are nested c...

  • 9965 Views
  • 7 replies
  • 3 kudos
Latest Reply
BilalAslamDbrx
Databricks Employee
  • 3 kudos

@Derk Crezee​ - I learned something today. Apparently ODBC does not convert to JSON. There is no defined spec on how to return complex types, in fact that was added only in SQL 2016. That's exactly what you are running into!End of history lesson Her...

  • 3 kudos
6 More Replies
RicksDB
by Contributor III
  • 5888 Views
  • 9 replies
  • 1 kudos

Configure jobs throttling for ephemeral cluster ETLs

Hi,Is it possible to configure job throttling in order to queue jobs across a workspace after a given number of concurrent execution when using the ephemeral cluster pattern? The reason is mainly for cost control. We prefer reducing performance rathe...

  • 5888 Views
  • 9 replies
  • 1 kudos
Latest Reply
RicksDB
Contributor III
  • 1 kudos

Thanks for the help josephk. I will continue to use an interactive cluster for the time being until the release of that new feature. Hopefully, it will allow my use case. Is there visibility on the roadmap for an ETA or more information on it?

  • 1 kudos
8 More Replies
barashe
by New Contributor II
  • 2098 Views
  • 1 replies
  • 0 kudos

Installing python modules on databricks job clusters

Different than all-purpose clusters, the databricks job new cluster configuration window does not have a "Libraries" tab, in which specific python modules could be installed. What's the best practice for installing python modules on such clusters?

  • 2098 Views
  • 1 replies
  • 0 kudos
Latest Reply
barashe
New Contributor II
  • 0 kudos

It turns out that the option exists outside of the cluster configuration scope, in the task configuration window itself - under "Advanced options" -> "Add dependent libraries".

  • 0 kudos
pthaenraj
by New Contributor III
  • 8589 Views
  • 10 replies
  • 14 kudos

Resolved! Databricks Certified Professional Data Scientist Exam Question Types

Hello,I am not seeing a lot of information regarding the Databricks Certified Professional Data Scientistexam. I took the Associate Developer in Apache Spark Exam last year and the materials for the exam seemed much more focused than what I found for...

  • 8589 Views
  • 10 replies
  • 14 kudos
Latest Reply
Abdull
New Contributor III
  • 14 kudos

Hello @Sundar R​ , Yes I took the exam. Unfortunately I fail to reach the pass mark even though I got close. Things I could have did different:I focused so much in mastering each topics i.e. linear, logistic & regularized regression, ALS and etc. But...

  • 14 kudos
9 More Replies
YSF
by New Contributor III
  • 3464 Views
  • 2 replies
  • 1 kudos

Resolved! Issues with using Databricks-Connect and Petastorm

Has anyone successfully used Petastorm + Databricks-Connect + Delta Lake?The use case is being able to use DeltaLake as a data store regardless of whether I want to use the databricks workspace or not for my training tasks.I'm using a cloud-hosted ju...

  • 3464 Views
  • 2 replies
  • 1 kudos
Latest Reply
YSF
New Contributor III
  • 1 kudos

because its janky or why? I don't need it for customer facing production. More so for if I'm using my own HPC or local workstation, but I want to access data from delta lake. Figured it was easier/preferable to setting up my own spark environment loc...

  • 1 kudos
1 More Replies
guruv
by New Contributor III
  • 21905 Views
  • 4 replies
  • 5 kudos

Resolved! parquet file to include partitioned column in file

HI,I have a daily scheduled job which processes the data and write as parquet file in a specific folder structure like root_folder/{CountryCode}/parquetfiles. Where each day job will write new data for countrycode under the folder for countrycodeI am...

  • 21905 Views
  • 4 replies
  • 5 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 5 kudos

Most external consumers will read partition as column when are properly configured (for example Azure Data Factory or Power BI).Only way around is that you will duplicate column with other name (you can not have the same name as it will generate conf...

  • 5 kudos
3 More Replies
Development
by New Contributor III
  • 1167 Views
  • 0 replies
  • 0 kudos

Hi All, I hope you're doing well I am facing issue while installing an python library on ADB Cluster. lib - PyCaret ( latest version) its not gett...

Hi All,I hope you're doing wellI am facing issue while installing an python library on ADB Cluster.lib - PyCaret ( latest version)its not getting install and showing me 'Failed' Status.It would be great if you can help here !!Thanks

  • 1167 Views
  • 0 replies
  • 0 kudos
TimK
by New Contributor II
  • 4791 Views
  • 2 replies
  • 1 kudos

Resolved! Cannot Get Databricks SQL to read external Hive Metastore

I have followed the documentation and using the same metastore config that is working in the Data Engineering context. When attempting to view the Databases, I get the error:Encountered an internal errorThe following information failed to load:The li...

  • 4791 Views
  • 2 replies
  • 1 kudos
Latest Reply
TimK
New Contributor II
  • 1 kudos

@Bilal Aslam​  I didn't think to look there before since I hadn't tried to run any queries. I see the failed SHOW DATABASES queries in history and they identify the error: Builtin jars can only be used when hive execution version == hive metastore v...

  • 1 kudos
1 More Replies
daschl
by Contributor
  • 15898 Views
  • 18 replies
  • 8 kudos

Resolved! NoSuchMethodError: org.apache.spark.sql.catalyst.json.CreateJacksonParser on Databricks Cloud (but not on Spark Directly)

Hi,I'm working for Couchbase on the Couchbase Spark Connector and noticed something weird which I haven't been able to get to the bottom of so far.For query DataFrames we use the Datasource v2 API and we delegate the JSON parsing to the org.apache.sp...

  • 15898 Views
  • 18 replies
  • 8 kudos
Latest Reply
daschl
Contributor
  • 8 kudos

Since there hasn't been any progress on this for over a month, I applied a workaround and copied the classes into the connector source code so we don't have to rely on the databricks classloader. It seems to work in my testing and will be released wi...

  • 8 kudos
17 More Replies
KaushalPatidar
by New Contributor II
  • 3659 Views
  • 3 replies
  • 0 kudos

I cannot access my account, please help

Hi, when I am trying to log into my account, its showing "invalid email address and password". But,I am sure everything is correct. I request @Kaniz Fatma​ , @Harikrishnan Kunhumveettil​ and @Prabakar Ammeappin​ to please look into it and resolve thi...

  • 3659 Views
  • 3 replies
  • 0 kudos
grandsurgical
by New Contributor
  • 693 Views
  • 0 replies
  • 0 kudos

Grand Surgical, established in 2010, has been manufacturing high-quality Surgical instruments for all disciplines of surgery. Cardiac, Vascular, denta...

Grand Surgical, established in 2010, has been manufacturing high-quality Surgical instruments for all disciplines of surgery. Cardiac, Vascular, dental, ophthalmic.We develop and deliver hospitals and medical professionals worldwide with superior qua...

  • 693 Views
  • 0 replies
  • 0 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels