cancel
Showing results for 
Search instead for 
Did you mean: 
Page Title

Welcome to the Databricks Community

Discover the latest insights, collaborate with peers, get help from experts and make meaningful connections

101679members
51739posts
cancel
Showing results for 
Search instead for 
Did you mean: 
Databricks Free Trial Feedback Event: AMA with Our Product Team

Hey Databricks Community, We're on a mission to make your free trial experience with Databricks even better, and we need your insights to do it! Whether you've dived into data engineering, explored data science, or analyzed analytics on our platform,...

  • 275 Views
  • 0 replies
  • 0 kudos
Wednesday
Announcing DBRX: A new standard for efficient open source LLMs

Are you ready to elevate your AI applications? Experience the power of DBRX, our cutting-edge open-source foundation model. Outperforming established open models in quality benchmarks, DBRX empowers you to swiftly craft bespoke LLMs tailored to your ...

  • 566 Views
  • 0 replies
  • 0 kudos
Wednesday
📢 Stay Updated with Databricks News!

Hey there, Data Enthusiasts! Curious about the latest buzz surrounding Databricks? Check out this insightful article from Fast Company, naming Databricks as one of the Most Innovative Companies of 2024!  How Databricks is helping customers develop t...

  • 620 Views
  • 1 replies
  • 4 kudos
a week ago
Session Catalog | Data + AI Summit 2024

Hey Data + AI Summit Community, We're thrilled to announce that the agenda for the upcoming Data + AI Summit is now live! Dive into the sessions, workshops, and events lined up for an unforgettable experience. To check out the latest session updates...

  • 1008 Views
  • 1 replies
  • 2 kudos
a week ago

Community Activity

AdamChan
by New Contributor
  • 182 Views
  • 3 replies
  • 0 kudos

The Voucher code provided to me by my organization is not working. Please help ASAP.

Hello,I have been recently given a voucher code by my organization to register for the Databricks Data Engineer Associate exam. However, when I tried to apply the code, the page kept showing "Candidate is not associated with the promotion code" despi...

  • 182 Views
  • 3 replies
  • 0 kudos
Latest Reply
Cert-Team
Honored Contributor III
  • 0 kudos

@AdamChan can you provide me with the ticket number and I will ask the support team. I reached out to them with the issue to get the ball rolling. Thanks!

  • 0 kudos
2 More Replies
sbs
by New Contributor II
  • 102 Views
  • 3 replies
  • 0 kudos

issue when reading csv in pandas

 Hello Team,I've encountered an issue while attempting to read a CSV data file into a pandas DataFrame by uploading it into DBFS in the community version of Databricks. Below is the error I encountered along with the code snippet I used:import pandas...

  • 102 Views
  • 3 replies
  • 0 kudos
Latest Reply
YuliyanBogdanov
New Contributor
  • 0 kudos

Assuming dbutils.fs.ls works without the "dbfs:/" prefix, try using it directly, i.e. df1 = pd.read_csv("/FileStore/shared_uploads/shiv/Dea.csv") . Alternatively, adjust the path as needed if using a local file path df1 = pd.read_csv("dbfs:/FileStore...

  • 0 kudos
2 More Replies
Ivan_
by Visitor
  • 24 Views
  • 0 replies
  • 1 kudos

Model logging issues - Azure Databricks

I am running job with service principal ( principal e1077c1b-1dcf-4975-8164-d14f748d4774). Mentioned job has a task that will log (mlflow.pyfunc.log_model) newly trained model to unity catalog, but I get PERMISSION DENIED error. I do not get why prin...

Screenshot 2024-03-29 100157.png Screenshot 2024-03-29 103021.png
  • 24 Views
  • 0 replies
  • 1 kudos
Jerry01
by New Contributor III
  • 15 Views
  • 0 replies
  • 0 kudos

How to override a in-built function in databricks

I am trying to override is_member() in-built function in such a way that, it always return true. How to do it in databricks using sql or python?

  • 15 Views
  • 0 replies
  • 0 kudos
Meshynix
by New Contributor II
  • 298 Views
  • 4 replies
  • 0 kudos

Not able to create external table in a schema under a Catalog.

Problem StatementCluster 1 (Shared Cluster) is not able to read the file location at "dbfs:/mnt/landingzone/landingzonecontainer/Inbound/" and hence we are not able to create an external table in a schema inside Enterprise Catalog.Cluster 2 (No Isola...

  • 298 Views
  • 4 replies
  • 0 kudos
Latest Reply
YuliyanBogdanov
New Contributor
  • 0 kudos

Hi @Meshynix,Can you provide the code snippet you execute to create your tables, this would give us a better insight for both use cases. Also can you provide the error that is being returned in the first use case. This would help a lot.

  • 0 kudos
3 More Replies
FlexException
by New Contributor II
  • 1090 Views
  • 5 replies
  • 0 kudos

Dynamic Number of Tasks in Databricks Workflow

Do Databricks workflows support creating a workflow with a dynamic number of tasks?For example, let's say we have a DAG like this:T1 ->    T2(1) ->             T2(2) ->              .....                 -> T3             T2(n-1) ->             T2(n)...

  • 1090 Views
  • 5 replies
  • 0 kudos
Latest Reply
tanyeesern
Visitor
  • 0 kudos

@FlexException Databricks API supports job creation and execution Task Parameters and Values in Databricks Workflows | by Ryan Chynoweth | MediumOne possibility is after running earlier job, process the output to create a dynamic number of tasks in s...

  • 0 kudos
4 More Replies
Pri333
by New Contributor II
  • 1209 Views
  • 10 replies
  • 1 kudos

Databricks exam got suspended. Need urgent help

Hello Team, I faced multiple interruptions and challenges while attempting my certification, The exam got paused 4 times where twice I had to show my desk and my entire room. Eventually my exam got suspended. I was not involved in any unfair means an...

  • 1209 Views
  • 10 replies
  • 1 kudos
Latest Reply
Cert-Team
Honored Contributor III
  • 1 kudos

@sai27 Support team has rescheduled your exam. Good luck! Please be sure to review the documentation to avoid getting suspended again: Understand the room requirementsUnderstand Behaviors to Consider when taking an online proctored exam

  • 1 kudos
9 More Replies
IshaBudhiraja
by New Contributor
  • 30 Views
  • 0 replies
  • 0 kudos

Migration of Synapse Data bricks activity executions from All purpose cluster to New job cluster

Hi,We have been planning to migrate the Synapse Data bricks activity executions from 'All-purpose cluster' to 'New job cluster' to reduce overall cost. We are using Standard_D3_v2 as cluster node type that has 4 CPU cores in total. The current quota ...

IshaBudhiraja_0-1711688756158.png
  • 30 Views
  • 0 replies
  • 0 kudos
superspan
by Visitor
  • 73 Views
  • 2 replies
  • 0 kudos

How to access Spark UI metrics in an automated way (API)

I am doing some automated testing; and would like ultimately to access per job/stage/task metrics as shown in the UI (e.g. spark UI -> sql dataframe) -> plan visualization in an automated way (API is ideal; but some ad-hoc metrics pipelines from loca...

  • 73 Views
  • 2 replies
  • 0 kudos
Latest Reply
superspan
Visitor
  • 0 kudos

Thanks for the response. This enables the event logs. But the event logs seem to be empty. Would you know where I can get the spark metrics as seen from the spark ui.

  • 0 kudos
1 More Replies
hossein_kolahdo
by New Contributor II
  • 149 Views
  • 2 replies
  • 0 kudos

Accessing data from a legacy hive metastore workspace on a new Unity Catalog workspace

Hello,For the purposes of testing I'm interested in creating a new workspace with Unity Catalog enabled, and from there I'd like to access (external - S3) tables on an existing legacy hive metastore workspace (not UC enabled). The goal is for both wo...

Data Engineering
hivemetastore
unitycatalog
Workspaces
  • 149 Views
  • 2 replies
  • 0 kudos
Latest Reply
hossein_kolahdo
New Contributor II
  • 0 kudos

@Kaniz From looking at the documentation none address my particular use case which I illustrated (2 workspaces on one account, 1 with UC and the other not). Was there a particular part on any of the docs you're suggesting can help here?

  • 0 kudos
1 More Replies
Geoff123
by New Contributor II
  • 198 Views
  • 8 replies
  • 0 kudos

Trouble on Accessing Azure Storage from Databricks (Python)

I used the same accessing method shown in https://community.databricks.com/t5/data-engineering/to-read-data-from-azure-storage/td-p/32230 but kept get the error below.org.apache.spark.SparkSecurityException: [INSUFFICIENT_PERMISSIONS] Insufficient pr...

  • 198 Views
  • 8 replies
  • 0 kudos
Latest Reply
Wojciech_BUK
Contributor III
  • 0 kudos

Hi,you can find storage account firewall information by accessing resource in azure portal Please mind that if you are using Unity Catalog you should NOT mount Storage Account, you should rather use abstraction of Storage Creadentials and External Lo...

  • 0 kudos
7 More Replies
Kayla
by Contributor
  • 62 Views
  • 0 replies
  • 0 kudos

Export notebook dashboard

I'm looking to find way to export notebook dashboards as HTML files.We will be scheduling the notebook via workflows, so I'm not sure if we'd be looking at exporting something from the workflow via API or if there's a better way to do this.I'm also c...

  • 62 Views
  • 0 replies
  • 0 kudos
Kibour
by New Contributor II
  • 465 Views
  • 1 replies
  • 0 kudos

Import from repo

Hi all,I am trying the new "git folder" feature, with a repo that works fine from the "Repos". In the new folder location, my imports from my own repo don't work anymore. Anyone faced something similar?Thanks in advance for sharing your experience

  • 465 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Kibour,  Ensure that the Databricks Repos feature is enabled. Sometimes, this issue arises when Repos are not properly activated.Additionally, verify that Repos are allowed for DBR 8.4+ versions.Confirm that your folder structure adheres to the c...

  • 0 kudos
Marinagomes
by Visitor
  • 138 Views
  • 1 replies
  • 0 kudos

raise Py4JJavaError while changing data type of a column

HiI'm using Azure databricks 10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12). I'm trying to convert 2 columns from string data type to timestamp data type . My date columns are in below format2/18/2021 7:20:12 PMSo I wrote following commandfrom py...

  • 138 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Marinagomes,  Try Using try_to_timestamp: Instead of to_timestamp, consider using try_to_timestamp. It returns null for malformed expressions, which can help identify problematic rows.

  • 0 kudos
Surajv
by New Contributor III
  • 81 Views
  • 1 replies
  • 0 kudos

What is the quota limit for using create user token api?

Hi Community, I was going through this doc: https://docs.databricks.com/api/workspace/tokens/create to and got to know, that there is a quota limit to how many token one can generate using the api: POST /api/2.0/token/create, having breached the thre...

  • 81 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Surajv, Let’s dive into the details of token quotas in Databricks. Quota Limit for Token Creation: The quota limit for creating user tokens via the API (specifically, using POST /api/2.0/token/create) is essential to manage token usage.Each u...

  • 0 kudos

Latest from our Blog

Maximizing Resource Utilisation with Cluster Reuse

Welcome to the fourth instalment of our blog series exploring Databricks Workflows, a powerful product for orchestrating data processing, machine learning, and analytics pipelines on the Databricks Da...

333Views 2kudos

MLOps Gym - Beginners Guide to MLflow

  MLflow MLflow stands out as the leading open source MLOps tool, and we strongly recommend its integration into your machine learning lifecycle. With its diverse components, MLflow significantly boo...

620Views 3kudos