cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Leladams
by New Contributor III
  • 13800 Views
  • 10 replies
  • 2 kudos

What is the best way to read in a ms access .accdb database into Databricks from a mounted drive?

I am currently trying to read in .accdb files from a mounted drive. Based on my research it looks like I would have to use a package like JayDeBeApi with ucanaccess drivers or pyodbc with ms access drivers.Will this work?Thanks for any help.

  • 13800 Views
  • 10 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Leland Adams​ Hope you are doing well. Thank you for posting your question and giving us additional information. Do you think you were able to solve the query?We'd love to hear from you.

  • 2 kudos
9 More Replies
DBU100725
by New Contributor II
  • 251 Views
  • 2 replies
  • 0 kudos

URGENT: Delta writes to S3 fail after workspace migrated to Premium

Delta writes to S3 fail after workspace migrated to Premium (401 “Credential was not sent or unsupported type”)SummaryAfter our Databricks workspace migrated from Standard to Premium, all Delta writes to S3 started failing with:com.databricks.s3commi...

  • 251 Views
  • 2 replies
  • 0 kudos
Latest Reply
dkushari
Databricks Employee
  • 0 kudos

Hi @DBU100725 - Are you using a No isolation shared cluster? Can you check if this was turned ON in your account?  

  • 0 kudos
1 More Replies
Shefali
by New Contributor
  • 374 Views
  • 1 replies
  • 0 kudos

Lakebridge conversion tool: Incorrect Databricks SQL script generated

Hi Team,I was able to successfully install and use the Lakebridge code conversion tool to convert my SQL Server script into a Databricks SQL script. However, the generated script contains several syntax errors. Could you please let me know if I might...

  • 374 Views
  • 1 replies
  • 0 kudos
Latest Reply
AbhaySingh
New Contributor
  • 0 kudos

Hi there!Known lakebase issues are listed here:https://github.com/databrickslabs/lakebridge/issuesDoes any of this apply to your use case?1. Variable scope errors in WHERE clauses or subqueries  2. DELETE/UPDATE FROM statements incorrectly converted ...

  • 0 kudos
devpavan
by Visitor
  • 17 Views
  • 2 replies
  • 0 kudos

Encountering an error while setting up a single-node cluster on top of aws

Hi Team,I'm trying to create a single-node cluster in Databricks on AWS, but I'm encountering an error. Could you please assist me with this?{ "reason": { "code": "INVALID_ARGUMENT", "type": "CLIENT_ERROR", "parameters": { "databr...

  • 17 Views
  • 2 replies
  • 0 kudos
Latest Reply
nayan_wylde
Honored Contributor III
  • 0 kudos

@devpavan Are you using API or terraform to create. Can you please share the json config that you are passing?

  • 0 kudos
1 More Replies
VaDim
by New Contributor III
  • 1726 Views
  • 1 replies
  • 1 kudos

ModuleNotFound error when using transformWithStateInPandas via a class defined outside the notebook

As per Databricks documentation when I define the class that extends `StatefulProcessor` in a Notebook everything works ok however, execution fails with ModuleNotFound error as soon as the class definition is moved to a file (module) of it's own in a...

Data Engineering
transformWithState
  • 1726 Views
  • 1 replies
  • 1 kudos
Latest Reply
stbjelcevic
Databricks Employee
  • 1 kudos

Hi @VaDim , Thanks for the detailed context — you’ve run into a common gotcha with how Python code is serialized and executed for stateful streaming on Databricks. Your sys.path.append only modifies the Python path on the driver node, but transformWi...

  • 1 kudos
Oumeima
by New Contributor
  • 1784 Views
  • 1 replies
  • 0 kudos

I can't use my own .whl package in Databricks app with databricks asset bundles

I am building a databricks app using databricks asset bundles. I need to use a helpers packages that i built as an artifact and using in other resources outside the app. The only way to use it is to have the built package inside the app source code f...

  • 1784 Views
  • 1 replies
  • 0 kudos
Latest Reply
stbjelcevic
Databricks Employee
  • 0 kudos

Hi @Oumeima , One potential way around this is to upload the wheel file into a Unity Catalog volume or workspace file. For the volume route, reference it directly in your app’s requirements.txt using an absolute /Volumes/<catalog>/<schema>/<volume>/....

  • 0 kudos
Davila
by New Contributor II
  • 1491 Views
  • 1 replies
  • 0 kudos

Issue with Root Folder Configuration in Databricks Asset Bundles for DLT Pipelines

I'm currently working with Databricks Asset Bundles to deploy my DLT pipelines, but I’ve encountered an issue I can't resolve.The problem is that I’m unable to configure the root folder within the Asset Bundle in a way that lets me define a custom pa...

Davila_1-1750959884979.png Davila_2-1750960256363.png
  • 1491 Views
  • 1 replies
  • 0 kudos
Latest Reply
Louis_Frolio
Databricks Employee
  • 0 kudos

Hey @Davila , I did some digging and have come with some things you can think about as you work through your issue. Here’s a clear way to think about what you’re seeing and how to proceed. What’s going on That “Root folder” field in the DLT UI is in...

  • 0 kudos
lauraxyz
by Contributor
  • 1972 Views
  • 6 replies
  • 0 kudos

Notebook in path workspace/repos/.internal/**_commits/** was unable to be accessed

I have a workflow job (source is git) to access a notebook and execute it.  From the job, it failed with error:Py4JJavaError: An error occurred while calling o466.run. : com.databricks.WorkflowException: com.databricks.NotebookExecutionException: FAI...

  • 1972 Views
  • 6 replies
  • 0 kudos
Latest Reply
lauraxyz
Contributor
  • 0 kudos

Just some clarification:  the caller notebook can be found with no issues, no matter the task's source is GIT or WORKSPACE.  However, the callee notebook, which is called by the caller notebook with dbutils.notebook.run(), cannot be found if the call...

  • 0 kudos
5 More Replies
JordanYaker
by Contributor
  • 2314 Views
  • 2 replies
  • 0 kudos

Integration options for Databricks Jobs and DataDog?

I know that there is already the Databricks (technically Spark) integration for DataDog. Unfortunately, that integration only covers the cluster execution itself and that means only Cluster Metrics and Spark Jobs and Tasks. I'm looking for somethin...

  • 2314 Views
  • 2 replies
  • 0 kudos
Latest Reply
greg-0935
Visitor
  • 0 kudos

Personally, I'm using their Data Jobs Monitoring product https://docs.datadoghq.com/data_jobs/databricks/ that works great and gives the right insights both for my high level job execution stats and Spark deeper metrics

  • 0 kudos
1 More Replies
Dhruv-22
by Contributor
  • 44 Views
  • 2 replies
  • 0 kudos

Resolved! Can't mergeSchema handle int and bigint?

I have a table which has a column of data type 'bigint'. While overwriting it with new data, given that I do full loads, I used 'mergeSchema' to handle schema changes. The new data's datatype was int. I thought mergeSchema can easily handle that, but...

  • 44 Views
  • 2 replies
  • 0 kudos
Latest Reply
Chiran-Gajula
  • 0 kudos

Hi Dhruv,Delta won't automatically upcast unless you explicitly handle it. Cast the column Lob_Pk to LongType (which maps to BIGINT in SQL/Delta). Try below snippetfrom pyspark.sql.functions import colfrom pyspark.sql.types import LongTypecrm_retail_...

  • 0 kudos
1 More Replies
saicharandeepb
by New Contributor III
  • 43 Views
  • 3 replies
  • 1 kudos

How to Retrieve DBU Count per Compute Type for Accurate Cost Calculation?

Hello Everyone,We are currently working on a cost analysis initiative to gain deeper insights into our Databricks usage. As part of this effort, we are trying to calculate the hourly cost of each Databricks compute instance by utilizing the Azure Ret...

  • 43 Views
  • 3 replies
  • 1 kudos
Latest Reply
nayan_wylde
Honored Contributor III
  • 1 kudos

1. Is there a documented way to retrieve the DBU count per VM or compute type?Yes, but it's not directly exposed via a single API or table. The DBU consumption rate depends on:Compute type (Jobs Compute, All-Purpose Compute, SQL Compute, etc.)VM inst...

  • 1 kudos
2 More Replies
Marthinus
by New Contributor III
  • 84 Views
  • 4 replies
  • 1 kudos

[Databricks Asset Bundles] Bug: driver_node_type_id not updated

Working with databricks asset bundles (using the new python-based definition), if you have a job_cluster defined using driver_node_type_id, and then update it to no longer have it defined, but only node_type_id, the driver node_type never gets update...

  • 84 Views
  • 4 replies
  • 1 kudos
Latest Reply
Chiran-Gajula
  • 1 kudos

There is no built-in way in Databricks Asset bundles or terraform to automatically inherit the value of driver_node_type_id for node_type_id."You must set both explicitly in your configuration"You can always see your updated detail resource from the ...

  • 1 kudos
3 More Replies
Dhruv-22
by Contributor
  • 1666 Views
  • 2 replies
  • 0 kudos

Resolved! Understanding least common type in databricks

I was reading the data type rules and found about least common type.I have a doubt. What is the least common type of STRING and INT? The referred link gives the following example saying the least common type is BIGINT.-- The least common type between...

  • 1666 Views
  • 2 replies
  • 0 kudos
Latest Reply
Dhruv-22
Contributor
  • 0 kudos

The question is solved here - link

  • 0 kudos
1 More Replies
Dhruv-22
by Contributor
  • 69 Views
  • 4 replies
  • 2 kudos

Resolved! Least Common Type is different in Serverless and All Purpose Cluster.

The following statement gives different outputs in different computes.In Databricks, 15.4 LTS%sqlSELECT typeof(coalesce(5, '6'));-- OutputstringIn Serverless, environment version 4%sqlSELECT typeof(coalesce(5, '6'));-- OutputbigintThere are other cas...

  • 69 Views
  • 4 replies
  • 2 kudos
Latest Reply
MuthuLakshmi
Databricks Employee
  • 2 kudos

@Dhruv-22 Regarding your 1st question, I'm not sureYou can refer to https://docs.databricks.com/aws/en/sql/language-manual/parameters/ansi_mode#system-default to understand what happens when ansi mode is disabled

  • 2 kudos
3 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels