I want to deploy a dabs that has dev using a github branch and prod using a github release tag. I can't seem to find a way to make this part dynamic based on the target. Things I've tried:- Setting the git varaible in the databricks.yml- making the g...
Hey, I'm Hasan Sayyed, currently pursuing SYBCA. I want to become a Data Engineer, but as a beginner, I’ve wasted some time learning other languages and technologies due to a lack of proper knowledge about this field. If someone could guide and teach...
Hi Hasan. Great to see your motivation! Here’s a good way to start your journey into data engineering:Master SQL, it’s the foundation of everything in data.Enroll in the Databricks Academy (free) and take the beginner courses like “Get Started with D...
Hey Community, I'm facing this error, It says that "com.databricks.pipelines.common.errors.deployment.DeploymentException: Communication lost with driver. Cluster 1030-205818-yu28ft9s was not reachable for 120 seconds" This issue occurred in producti...
This is actually a known intermittent issue in Databricks, particularly with streaming or Delta Live Tables (DLT) pipelines.This isn’t a logical failure in your code — it’s an infrastructure-level timeout between the Databricks control plane and the ...
1. How could I pull workspace url (https://adb-XXXXX.XX.....net) 2. How could I get workspace name visible in top right corner.I know that easies solution is dbutils.notebook.entry_point.... browserHostName but unfortunetly it is not working in job c...
Can you give this a shot? Not sure if you've a hard requirement of using SDK. workspace_url = spark.conf.get('spark.databricks.workspaceUrl') Getting name is more tricky. You could potentially get it from tags if there is a tagging strategy in place...
We are wanting to use Databricks asset bundles and deploy code changes and tests using github actions. We have seen lots of content online, but nothing concrete on how this is done at scale. So I'm wondering, if we have many changes and therefore man...
Have you read about following approach before? Repository Structure Options 1. Monorepo with Multiple Bundles repo-root/ ├── .github/ │ └── workflows/ │ ├── bundle-ci.yml │ └── bundle-deploy.yml ├── bundles/ │ ├...
Hi all,I need some help on this masking problem. If you create a view with used masking function based on table.The user reading this view has to have read access to underlying table. So theoretically, he can access unmasked data in the table.I would...
Are you on Unity catalog? Databricks has a solution for this through Unity Catalog Column Masking (also called Dynamic Views or Column-Level Security). https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/filters-and-mask...
Looking for example databricks.yml and bundle resources for deploying a FastAPI Databricks app using DABs from one environment to another. Deployment works but FastAPI redirects to localhost after deployment, though the homepage loads fine. Need refe...
This is a great place to start: https://apps-cookbook.dev/resources/ Happy to answer specifics as they come after you've reviewed that resource.
I have a Repo in Databricks connected to Azure DevOps Repositories.The repo has been working fine for almost a month, until last week. Now when I try to open the Git settings in Databricks, I am getting "Invalid Git Credentials". Nothing has change...
I had a similar problem. I could fix following these steps:in the Azure Devops repository: User Settings -> Personal access tokens -> + New tokenin Databricks: Settings -> User -> Linked accounts -> Azure Devops (Personal access token)You could also...
Since there is no way to file a bug, I'll post it here.. Honestly, I haven't seen such a broken and inconsistent API from production system yet in my life..what is worse - this same issue is in 'os' module:And their UI (despite actually showing the f...
Hi @whatever, Thanks for sharing this. I will test this and report internally, meanwhile you can also submit a new idea/request/bug using this portal from your end: https://docs.databricks.com/en/resources/ideas.html#create-an-idea-in-the-ideas-port...
Hello everyone,I’m in the process of using Blade Bridge to convert my SSIS .dtsx packages into Databricks SQL, but I’ve run into a licensing issue and could use some guidance.What I’m doing:Installed Blade Bridge and followed the required folder stru...
Hi @Rainier_dw @Eric_Kieft , https://github.com/databrickslabs/lakebridge/issues/1819 is now tracked under https://github.com/databrickslabs/lakebridge/issues/1836 as an enhancement for product and https://github.com/databrickslabs/lakebridge/pull/1...
Hi Team , I want to take advantage of Spark Distribution over GPU clusters using RAPID(NVIDIA) , everything is setup 1. The Jar is loaded correctly via Init script , the jar is downloaded and uploaded on volume (workspace is unity enabled) and via In...
Also try to gradually reduce spark.executor.memory You need to allocate less memory to the JVM heap because the GPU needs a large chunk of the node's off-heap (system) memory. The GPU memory is allocated outside the JVM heap. If the heap is too large...
Thanks for the Databricks community and maintaining such a valuable platform.I would like to inquire if there is a planned timeline for upgrading the GraphFrame library. We’ve noticed that the latest release on GitHub is v0.9.3, while the Databricks ...
Greeting @toproximahk , thanks for the kind words and for the detailed pointers. What’s in Databricks Runtime 17.3 LTS ML today The preinstalled GraphFrames JAR in Databricks Runtime 17.3 LTS for Machine Learning is org.graphframes:graphframes_2.1...
Databricks DLT Joins: Streaming table join with Delta table is reading 2 Billion records from Delta Table for each and every Micro batch.How to overcome this issue to not to read 2 Billion records for every micro batch.Your suggestions and feedback w...
I am developing a Databricks application using the Streamlit package. I was able to get a "hello world" app deployed successfully, but now I am trying to pass data that exists in the dbfs on the same instance. I try to read a csv saved to the dbfs bu...
I used Azure Blob Storage, and this resolved the issue. Our app now contains Python files in the Databricks environment app that have access to Azure Blob Storage using Azure credentials.
Today, while reading a delta load my notebook failed and I wanted to report a bug. The withColumns command does not tolerate an empty dictionary and gives the following error in PySpark.flat_tuple = namedtuple("flat_tuple", ["old_col", "new_col", "lo...
Hello @Dhruv-22 , I have tested this internally, and this seems to be a bug with the new Serverless env version 4 As a solution, you can try switching the version to 3 as shown bleow and re-run the above code, and it should work.
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 483 | |
| 349 | |
| 287 |
