cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

lauraxyz
by New Contributor III
  • 30 Views
  • 2 replies
  • 0 kudos

dbutils.notebook API: pass data back to caller notebook

Hi all, according to this doc, we can pass data back through temp views, DBFS, or JSON data.However, in my case, i need to pass both a temp view, as well as some metadata in JSON.  is there a way to exit with BOTH a view AND json, something likedbuti...

  • 30 Views
  • 2 replies
  • 0 kudos
Latest Reply
lauraxyz
New Contributor III
  • 0 kudos

I can have a try and see if it's the same like exit(view_name) in that the view is created in global_temp_db and that the lifecycle is with the job compute.

  • 0 kudos
1 More Replies
dixonantony
by New Contributor III
  • 466 Views
  • 8 replies
  • 0 kudos

Not able create table form external spark

py4j.protocol.Py4JJavaError: An error occurred while calling o123.sql.: io.unitycatalog.client.ApiException: generateTemporaryPathCredentials call failed with: 401 - {"error_code":"UNAUTHENTICATED","message":"Request to generate access credential for...

  • 466 Views
  • 8 replies
  • 0 kudos
Latest Reply
NandiniN
Databricks Employee
  • 0 kudos

You need the generateTemporaryPathCredentials API as you are trying to create external tables 

  • 0 kudos
7 More Replies
eballinger
by New Contributor III
  • 40 Views
  • 3 replies
  • 1 kudos

Resolved! DLT notebook dynamic declaration

Hi Guys,We have a DLT pipeline that is reading data from landing to raw (csv files into tables) for approximately 80 tables. In our first attempt at this we declared each table separately in a python notebook. One @Dlt table declared per cell. Then w...

  • 40 Views
  • 3 replies
  • 1 kudos
Latest Reply
VZLA
Databricks Employee
  • 1 kudos

Hello @eballinger , thank you for your question. To better assist you, could you clarify a few details? Are you seeing delays in specific stages (e.g., metadata fetching, schema validation, or table setup)?Could you provide more details on how the dy...

  • 1 kudos
2 More Replies
jeremy98
by Contributor
  • 770 Views
  • 3 replies
  • 0 kudos

token share

Hello community,I want to have a new token, to be available to our users to interact with staging workspace. Is it possible to generate a token to be used for triggering only workflow in staging workspace databricks with databricks api?

  • 770 Views
  • 3 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

you will need to have your own personal token to create one for your SP, in the UI you need to go to Settings > Under User > Developer > Personal Access Token.Once you have your own token you can run the API I mentioned on my previous post and you ne...

  • 0 kudos
2 More Replies
TejeshS
by New Contributor II
  • 15 Views
  • 1 replies
  • 0 kudos

Event based Alert based on certain events from System Audit tables

We need to implement an event-based trigger system that can detect any manual intervention performed by users. Upon detection of such an event, the system should automatically send a warning email. The events can be generated through DLT or other pro...

  • 15 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Just to understand your request you are looking if there is another way to get users events outside the system tables that can make your workflow more efficient?  

  • 0 kudos
Paul92S
by New Contributor III
  • 129 Views
  • 7 replies
  • 4 kudos

Delta sharing service Issue making requests to Unity System Access tables

Hi all, We have been having an issue as of yesterday which I believe is related to queries against the system.access.table_linage in Unity Catalogs. This issue still persists todayWe get the following error:AnalysisException: [RequestId= ErrorClass=B...

table lineage.png delta sharing issue.png
  • 129 Views
  • 7 replies
  • 4 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 4 kudos

Team, I will provide an update as soon as we hear from our backend team working on this. 

  • 4 kudos
6 More Replies
jeremy98
by Contributor
  • 60 Views
  • 5 replies
  • 1 kudos

For each task field

Hi community,I was wondering after passing a list of dict through tasks using .taskValue.set() method, how to maintain the same data type through each task?Because seems, that when I use the for loop and getting by the parameters each element of the ...

  • 60 Views
  • 5 replies
  • 1 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 1 kudos

Yeah, to ensure that the data types are maintained, you can convert the values to the desired types after deserialization. This is necessary because JSON does not distinguish between integers and floats, and all numbers are deserialized as floatsThe ...

  • 1 kudos
4 More Replies
Rjdudley
by Contributor II
  • 19 Views
  • 1 replies
  • 0 kudos

Deploying Data Source API code

This might be a stupid question but there's just no mention of what to do here.  I'm looking at the blog (https://www.databricks.com/blog/simplify-data-ingestion-new-python-data-source-api) and documentation (https://learn.microsoft.com/en-us/azure/d...

  • 19 Views
  • 1 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hi @Rjdudley, Thanks for your question - You can create regular .py files in your workspace and use the %run magic command to include them in your notebooks. This method is straightforward and good for development and testing. %run /path/to/your/cust...

  • 0 kudos
VJ3
by Contributor
  • 1196 Views
  • 3 replies
  • 0 kudos

Databricks Upload local files (Create/Modify table)

Hello Team,I believe Databricks come out recently feature of Create or modify a table using file upload which is less than 2 GB (file format CSV, TSV, or JSON, Avro, Parquet, or text files to create or overwrite a managed Delta Lake table) on Self Se...

  • 1196 Views
  • 3 replies
  • 0 kudos
Latest Reply
NandiniN
Databricks Employee
  • 0 kudos

For Sharing a CSV file containing PII data with another user who should not have access to PII data elements: You can use Databricks' Unity Catalog to manage and govern access to data. Unity Catalog allows you to define fine-grained access controls a...

  • 0 kudos
2 More Replies
alpar
by Visitor
  • 78 Views
  • 4 replies
  • 4 kudos

Merge operation to delta table with new column starting with upper case seems to be not working

Hello,I have a simple spark dataframe saved to a delta table:data = [ (1, "John", "Doe"), (2, "Jane", "Smith"), (3, "Mike", "Johnson"), (4, "Emily", "Davis")]columns = ["Id", "First_name", "Last_name"]df = spark.createDataFrame(data, sche...

  • 78 Views
  • 4 replies
  • 4 kudos
Latest Reply
hari-prasad
Valued Contributor
  • 4 kudos

I assume you must be facing an error referred here on GitHub issues page. you can follow it, they make release fix for same.[BUG][Spark] issue when merge using autoMerge property · Issue #3336 · delta-io/delta · GitHub

  • 4 kudos
3 More Replies
REM1992
by New Contributor
  • 71 Views
  • 1 replies
  • 0 kudos

Alert monitoring, not running in schedule

Hello, I think the alert that I set is not running on the schedule that I set , every day 9 am JST time. It shows up like it is running, with the symbol of running moving , but it says since 2025/1/7 while it should have been run at 2025/1/8 9:00 am ...

  • 71 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

The day showing there is the day since the first time it was executed, if you check on the job runs do you see that there are jobs running every day on that period of time?

  • 0 kudos
mh7
by New Contributor
  • 59 Views
  • 3 replies
  • 0 kudos

spark throws error while using [NOT_IMPLEMENTED] rdd is not implemented.

i am running code in 15.4lts and it works fine in all purpose cluster.processed_counts = df.rdd.mapPartitions(process_partition).reduce(lambda x, y: x + y)when i run the same code using job cluster, it throw's below error. I verfied the cluster setti...

  • 59 Views
  • 3 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Ok, but your all purpose cluster is set up with Single User mode which is indeed supported for the RDD, can you confirm your job cluster is also created by using Single user mode?

  • 0 kudos
2 More Replies
stevomcnevo007
by New Contributor II
  • 337 Views
  • 13 replies
  • 2 kudos

agents.deploy NOT_FOUND: The directory being accessed is not found. error

I keep getting the following error although the model definitely does exist and version names and model name is correct RestException: NOT_FOUND: The directory being accessed is not found. when calling # Deploy the model to the review app and a model...

  • 337 Views
  • 13 replies
  • 2 kudos
Latest Reply
ezermoysis
New Contributor
  • 2 kudos

@Alberto_UmanaBelow I add all the log:File <command-3498969875739223>, line 12 10 print(UC_MODEL_NAME) 11 print(uc_registered_model_info.version) ---> 12 deployment_info = agents.deploy("emoysis_catalog.cookbook.sop_chatbot", 13) 14 # ...

  • 2 kudos
12 More Replies
Databricks_-Dat
by New Contributor II
  • 6310 Views
  • 4 replies
  • 1 kudos

Databricks workflows, sample script/method to deploy jobs.json to other workspace

Could someone point me at right direction to deploy Jobs from one workspace to other workspace using josn file in Devops CI/CD pipeline? Thanks in advance.

  • 6310 Views
  • 4 replies
  • 1 kudos
Latest Reply
yuvapraveen_k
New Contributor III
  • 1 kudos

Your are welcome. There was a feature that databricks released to linked the workflow definition to the GIT automatically. Please refer the link below,https://www.databricks.com/blog/2022/06/21/build-reliable-production-data-and-ml-pipelines-with-git...

  • 1 kudos
3 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels