Join the Databricks Learning Festival (Virtual)! Mark your calendars from 10 October - 31 October 2024! Upskill today across data engineering, data analysis, machine learning, and generative AI. Join the thousands who have elevated their career w...
- 74271 Views
- 367 replies
- 53 kudos
09-17-2024
What better way to enjoy the fall and prepare for the holidays than getting free Databricks training? Join us for a Databricks Hybrid Learning Day in New York City (in person) and get an opportunity to work on self-paced Databricks training (on Datab...
- 535 Views
- 0 replies
- 0 kudos
3 weeks ago
Migrating your data warehouse workloads is one of the most challenging yet essential tasks for any organization. Whether the motivation is the growth of your business and scalability requirements or reducing the high license and hardware cost of your...
- 194 Views
- 0 replies
- 1 kudos
Thursday
Over the past few months, we’ve been gathering your feedback and focusing on both the quality of Databricks Assistant’s responses and the overall user experience. Today, we're excited to showcase a more advanced Databricks Assistant, packed with powe...
- 220 Views
- 0 replies
- 1 kudos
Thursday
Over the years, organizations have amassed a vast amount of unstructured text data—documents, reports, and emails—but extracting meaningful insights has remained a challenge. Large Language Models (LLMs) now offer a scalable way to analyze this data,...
- 212 Views
- 0 replies
- 0 kudos
Thursday
Community Activity
- 0 Views
- 0 replies
- 0 kudos
Late file arrivals - Autoloader
Hi All,I have a situation where I'm receiving various CSV files in a storage location.The issue I'm facing is that I'm using Databricks Autoloader, but some files might arrive later than expected. In this case, we need to notify the relevant team ab...
- 0 Views
- 0 replies
- 0 kudos
- 40 Views
- 2 replies
- 1 kudos
cloud_infra_costs
I was looking at the system catalog and realized that there is an empty table called cloud_infra_costs. Could you tell me what is this for and why it is empty?
- 40 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Thanks for replying. This makes sense. Any idea why it is empty and what to do to populate this?
- 1 kudos
- 20 Views
- 3 replies
- 0 kudos
Restricting access to secrets
Hi. I want to restrict access to secrets to a security group, as the secrets can be used to retrieve sensitive data only a few people should see. Up until now, we have been using KV-backed secret scopes, but as it's sufficient that Databricks has the...
- 20 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi Johan, this should work for restriction: https://learn.microsoft.com/en-us/azure/databricks/security/secrets/secrets.Fine granulat access based on secrets is currently not possible.BR
- 0 kudos
- 1554 Views
- 2 replies
- 0 kudos
Parameterized Delta live table pipeline
I'm trying to create an ETL framework on delta live tables and basically use the same pipeline for all the transformation from bronze to silver to gold. This works absolutely fine when I hard code the tables and the SQL transformations as an array wi...
- 1554 Views
- 2 replies
- 0 kudos
- 13 Views
- 0 replies
- 0 kudos
Automating Enterprise Application Setup for Azure Databricks SCIM Provisioning
Hello Databricks Community,We are working on automating the creation of an Enterprise Application for the Azure Databricks SCIM Provisioning Connector, with the goal of enabling automated user and group provisioning from Azure Active Directory (Azure...
- 13 Views
- 0 replies
- 0 kudos
- 8 Views
- 0 replies
- 0 kudos
Table-Model Lineage for models without online Feature Lookups
Hi community,I am looking for the recommended way to achieve table-model lineage in Unity Catalog for models that don't use Feature Lookups but only offline features. When I use FeatureEngineeringClient.create_training_set with feature_lookups + mlfl...
- 8 Views
- 0 replies
- 0 kudos
- 7 Views
- 0 replies
- 0 kudos
Exam got suspended
Hi on Sunday at 13:00 Indian time when I was giving my Databrick Certified Dat Engineering Associate certfication . My exam got suspended with a warning indicating there was some environmental behavioral issue. But I didnt do any thong wrong and I ha...
- 7 Views
- 0 replies
- 0 kudos
- 15 Views
- 0 replies
- 0 kudos
Enabling Object Lock for the S3 bucket that is delivering audit logs
Hello Community,I am trying to enable Object Lock on the S3 bucket to which the audit log is delivered, but the following error occurs if Object Lock is enabled when the delivery settings are enabled.> {"error_code":"PERMISSION_DENIED","message":"Fai...
- 15 Views
- 0 replies
- 0 kudos
- 17 Views
- 0 replies
- 0 kudos
Error connecting to MySQL from Databricks: (2003, "Can't connect to MySQL server")
Hello Community,I'm facing an issue connecting to a MySQL database hosted on AWS RDS from within a Data bricks notebook. My Python script to connect to MySQL works fine locally, but when I run it in Data bricks, I receive this error:Error connecting ...
- 17 Views
- 0 replies
- 0 kudos
- 23 Views
- 0 replies
- 0 kudos
We are trying to connect to AWS RDS MySQL instance from DBX with PySpark using JDBC
We are trying to connect to AWS RDS MySQL instance from DBX with PySpark using JDBC: jdbc_df = (spark.read.format("jdbc").options(url=f"jdbc:mysql://{creds['host']}:{creds['port']}/{creds['database']}", driver="com.mysql.cj.jdbc.Driver", dbtable="(SE...
- 23 Views
- 0 replies
- 0 kudos
- 32 Views
- 1 replies
- 0 kudos
"mlflow.login()" not working with "https://community.cloud.databricks.com"
Hi,I'm trying to use databricks community edition with mlflow, and I tried to follow the instructions from https://www.databricks.com/blog/2019/10/17/managed-mlflow-now-available-on-databricks-community-edition.htmlIt looks like that "mlflow.login()"...
- 32 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Also, the instructions from https://docs.databricks.com/en/mlflow/access-hosted-tracking-server.html
- 0 kudos
- 45 Views
- 1 replies
- 0 kudos
Plans to upgrade API to get dashboard details for non-legacy dashboards?
I'm interested in using the API to list all dashboards, but the API shows legacy dashboards only. Are there any plans to fix this API call to show all dashboards?I'm using this API call: https://docs.databricks.com/api/workspace/dashboards/listAnd th...
- 45 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Follow up...this API to get dashboard details seems to work only for legacy dashboards: https://docs.databricks.com/api/workspace/dashboards/get
- 0 kudos
- 98 Views
- 1 replies
- 0 kudos
SparkRuntimeException: [UDF_ERROR.ENV_LOST] the execution environment was lost during execution
Hey everyone,I have been facing a weird error when i upgrade to use Unity Catalog.org.apache.spark.SparkRuntimeException: [UDF_ERROR.ENV_LOST] Execution of function line_string_linear_interp(geometry#1432) failed - the execution environment was lost ...
- 98 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @calvinchan_iot, How are you doing today?As per my understanding, It sounds like the error may be due to environment instability when running the UDF after enabling Unity Catalog. The [UDF_ERROR.ENV_LOST] error often points to the UDF execution en...
- 0 kudos
- 77 Views
- 1 replies
- 0 kudos
Schema evolution in Autoloader not evolving beyond version 0
I am working through the current version of the standard AutoLoader demo, i.e. dbdemos.install('auto-loader')I.e. data gets read into a dataframe, but never written to a target table.Notebook is "01-Auto-loader-schema-evolution-Ingestion"Compute i...
- 77 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @RoelofvS,How are you doing today? As per my understanding, You just make sure cloudFiles.schemaEvolutionMode is set to addNewColumns to enable automatic schema updates for new columns. If schema versions aren't updating in the same location, try ...
- 0 kudos
- 17842 Views
- 24 replies
- 4 kudos
Suspended Exam? Don't worry! Our support team can help!
Taking exams is so stressful already, and if your exam gets suspended it is extra stressful. Don't worry! Our support team is awesome and will help you get back on track. The most important step is to file a ticket in our Help Center. Be sure to inc...
- 17842 Views
- 24 replies
- 4 kudos
Latest Reply
- 4 kudos
Hello @Cert-Team Raised ticket Number:#00557482My exam got suspended due to noise from outside area of office. I was reviewing the marked question before I submit the exam but it suddenly suspended saying that environmental violation and behavioural ...
- 4 kudos
Announcements
Latest from our Blog
Efficiently migrating SQL script backups from legacy databases with Databricks and DuckDB
For many use cases, the de facto method for loading tabular data into a Databricks Lakehouse from a relational database (RDBMS) uses an ETL tool to connect using JDBC or CDC. The data is read, frequen...
Data Cleaning for Machine Learning
Data cleaning is an essential data preprocessing step in preparing data for machine learning. The quality of data directly impacts model performance, and these processes ensure that the data is accura...
💡 ML Training Tip Of The Week #3 - Adjust Sample Weight for Imbalanced Dataset in AutoML
When running an AutoML experiment on Databricks, the default setup treats each data sample as equally important. However, this approach can be problematic when dealing with highly imbalanced datasets....
High Performance Streaming from Azure Event Hubs using Apache Spark
by Peter Stern & Sanja Sandic A couple of years ago, we, two Solutions Architects at Databricks, were working with a customer to maximize the performance when reading from Azure Event Hubs into Datab...
