- 1042 Views
- 3 replies
- 1 kudos
Resolved! Schema Evolution and Schema Enforcement without Delta live Tables & Unity catalog
In Delta Lake, schema evolution with mergeSchema handles column additions perfectly — new columns get added and old rows get NULL. But when there is a data type change in the incoming data (for example, a column that was INT now coming as STRING from...
- 1042 Views
- 3 replies
- 1 kudos
- 1 kudos
Yes — defining the schema manually for production is okay when you are not using Auto Loader. With a manually provided schema, you should expect stricter enforcement: Delta will not automatically absorb non-widening type changes like INT -> STRING in...
- 1 kudos
- 4250 Views
- 1 replies
- 1 kudos
Seeking GitBook Alternatives for Documenting Databricks Pipelines & ML Workflows
We’re scaling our use of Databricks notebooks and ML pipelines, and keeping everything properly documented is becoming a challenge. I’ve used GitBook in past projects, but it’s starting to feel a bit clunky, especially when trying to keep markdown do...
- 4250 Views
- 1 replies
- 1 kudos
- 1 kudos
There are are a bunch of choices for API documentation tools, especially if you are comfortable with markdown or MDX, GitHub, etc.Some to consider other than GitBook are: Mintlify, Readme, Docusaurus (self-hosted), and Jamdesk.
- 1 kudos
- 398 Views
- 3 replies
- 0 kudos
Databricks missing scheduler for scheduled jobs
Suppose a job is scheduled at 2 am but if scheduler misses it we need to get an alert message how can it be possible.we don't have centralised table monitoring and APIs for it
- 398 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Teja3051, Yes, it is possible for a scheduled run to be delayed or skipped, so I’d first recommend identifying the root cause before putting an alerting workaround in place. Common things to check are scheduler or cloud delays of a few minutes, m...
- 0 kudos
- 350 Views
- 1 replies
- 2 kudos
What I’m Seeing in the Databricks World Right Now
The energy around Databricks feels different this year. It’s no longer just about faster queries or better pipelines. It’s about building intelligent systems on top of trusted data. The conversations have shifted. The expectations have shifted. Even ...
- 350 Views
- 1 replies
- 2 kudos
- 2 kudos
I totally agree with you !I have always learnt by joining communities
- 2 kudos
- 541 Views
- 2 replies
- 3 kudos
Resolved! not able to access "Compute"
when i click on Compute it redirects me to the SQL warehouse section. Why is that happening? I believe i have the enterprise plan with pay as you go enabled.
- 541 Views
- 2 replies
- 3 kudos
- 3 kudos
I am having the same problem. Signed up for a new account (separate from the free trial account) under the same email address. Added credit card details. But I get redirected to SQL Warehouses when I click on Compute.Workspace ID 7474652800168347
- 3 kudos
- 838 Views
- 4 replies
- 0 kudos
Resolved! Databricks Platform Administrator
I am an Informatica Administrator. Want to become Databricks Administrator. Would like to connect with people who are working as Databricks platform admins and understand the path to become a successful Admin.
- 838 Views
- 4 replies
- 0 kudos
- 0 kudos
I think the slides are there for reference . There should be some video later that references that. But I understand that can be be confusing . The notebooks would be downloadable from another version of the course that has a paid labs . So if you ar...
- 0 kudos
- 322 Views
- 1 replies
- 1 kudos
Resolved! Requesting for assistance in creating a user group for my region
Hello Databricks team, I hope you are doing well. I am based in Boston and currently working as a manager in healthcare analytics with one of the Databricks partners. I have been conducting Databricks events and meetups in Boston as well as NY but we...
- 322 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi !You can submit your application here :https://community.databricks.com/t5/announcements/the-new-databricks-user-groups-platform-is-here/td-p/146919
- 1 kudos
- 3525 Views
- 2 replies
- 0 kudos
Archival Strategy for Delta tables
Hi Team, We would like to define the archival strategy for data. Could you please share best practices /guide me on the below are the 3 use cases Case-1: On-Prem SQL and Oracle Data which is more than 20 years and they wanted to bring them into clou...
- 3525 Views
- 2 replies
- 0 kudos
- 0 kudos
case 1: I'd extract the data from the db to a data lake (cold storage if that is possible, that is cheaper) using an ETL tool like Data Factory, Glue etc. Then the archiving can take place. Perhaps also create a backup of the data on a 2nd data lak...
- 0 kudos
- 1553 Views
- 4 replies
- 1 kudos
Databricks Java SDK retrieving job task values
Greetings,I have a Job that consists of notebook tasks running python code.Some of the task set task values using dbutils.jobs.taskValues.set(key=key, value=value)as described here How do I retrieve those task values using Databricks Java SDK v0.69.0...
- 1553 Views
- 4 replies
- 1 kudos
- 1 kudos
Unfortunately you can’t read dbutils.jobs.taskValues directly via the Java SDK. But you could return a JSON payload via dbutils.notebook.exit and read it with getRunOutput. Databricks exposes the notebook “exit” result through Jobs GetRunOutput, then...
- 1 kudos
- 521 Views
- 3 replies
- 1 kudos
Lakeflow jobs file trigger thru overwritten files
Is there any way a job can be triggered with overwritten files through a file trigger?I knew the limitations, but I just want to have other opinions or solutions.LimitationsOnly new files trigger runs. Overwriting an existing file with a file of the ...
- 521 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @DazzaiDe, Unfortunately, you can't do that as of today. Is there a specific reason you want to maintain the same name for the triggers? Instead of linking to a static file name, could you point to a root path so that any new files will trigger th...
- 1 kudos
- 279 Views
- 1 replies
- 0 kudos
Error while reading file deletion_vector_5274904e-d64e-44e9-9a4b-1d7e93dfdc0b.bin
org.apache.spark.sql.streaming.StreamingQueryException: [STREAM_FAILED] Query [id = c981c01f-d84b-4f2b-8b78-c7324cd7e752, runId = 7bdd53a0-0d2b-4456-aad8-734e82116c1d] terminated with exception: [FAILED_READ_FILE.NO_HINT] Error while reading file abf...
- 279 Views
- 1 replies
- 0 kudos
- 0 kudos
Tough to say from just this. You should create a support ticket with MSFT. Possible the deletion vector file was deleted out from under Auto Loader mid-read. Which might be a bug or it could be that you have some separate cleanup running at the same ...
- 0 kudos
- 1027 Views
- 1 replies
- 0 kudos
Resolved! Best practices for using autoloader
I’m looking to follow best practices with Databricks Auto Loader. When ingesting different file formats, is it considered good practice to always read data as strings first, or is it better to use format-specific readers (e.g., JSON, CSV, binary) fro...
- 1027 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @DazzaiDe , Use the format-specific readers from the start and let Auto Loader handle schema, rather than reading everything as generic strings/text yourself. Key points: Always set cloudFiles.format to the real file format (json, csv, xml, parqu...
- 0 kudos
- 541 Views
- 1 replies
- 1 kudos
workspace config
"I am on a Premium AWS trial workspace (dbc-30503d28-2210). I have two issues:Personal access tokens are grayed out and I cannot generate themMy cluster cannot make outbound HTTP requests to external APIs (getting NameResolutionError when calling api...
- 541 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi, On the personal access tokens greyed out, you need to enable this in the workspace settings. (you'll need to make sure you're a workspace admin first). This is in the advanced setting under workspaces ettings. Are you validating you're a workspa...
- 1 kudos
- 2968 Views
- 1 replies
- 3 kudos
2026 update: Postman free plan limits and alternatives?
Hey,I’ve been rethinking my API tooling lately and realized I’ve mostly stayed with Postman out of habit.One thing that stood out again is the free plan limitations. They’re not new, but they make collaboration a bit annoying for small teams unless y...
- 2968 Views
- 1 replies
- 3 kudos
- 3 kudos
Hi @john26 ,I use Bruno quite often these days and it become the go-to Postman replacement for engineering-heavy workflows specifically because collections live as plain files in your repo. For someone working across Databricks, ADF, and Azure servic...
- 3 kudos
- 577 Views
- 3 replies
- 0 kudos
Resolved! RESOURCE_EXHAUSTED - I cannot work with a serverless in my free account. Do you know why?
Hi team, I am seeing this when trying to have a dummy project. Do you know why?
- 577 Views
- 3 replies
- 0 kudos
- 0 kudos
@alfredoigmen Databricks workspace's compute resources will be shut down and unavailable for the rest of the day (a month in extreme cases) if you exceed the quota. While the compute will be unavailable, Data and settings will not be deleted. You can...
- 0 kudos
