- 5057 Views
- 12 replies
- 17 kudos
Resolved! Accessing Spark UI in free edition
Hello, is it possible to access Spark UI in free edition, I want to check task and stages.Ultimately I am working on how to check data skewness.
- 5057 Views
- 12 replies
- 17 kudos
- 17 kudos
@szymon_dybczak @BS_THE_ANALYST Is there a specific guide or a flow to be a better databricks data engineer. I am learning as the topic comes up.Finding it really difficult to maintain a flow and I lose track.
- 17 kudos
- 770 Views
- 2 replies
- 1 kudos
Resolved! Unable to Verify Account – "User is not a member of this workspace" Community addition..
I am encountering an issue during the verification process while trying to access my Databricks account.I received the verification code via email; however, when I attempt to verify it, I receive the following error message:"User is not a member of t...
- 770 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello @perenehal! The error indicates that you haven’t created an account in the Databricks Community Edition. Currently, creating new accounts for Community Edition is not possible. However, if you already had an existing account and are still seein...
- 1 kudos
- 7807 Views
- 2 replies
- 1 kudos
How agentic ai is different from Ai agents
How agentic ai different from ai agents
- 7807 Views
- 2 replies
- 1 kudos
- 1 kudos
Hey there! Let me break this down simply.What's an AI Agent?An AI agent is like a smart worker who follows instructions for one specific job. Think of a customer service chatbot that answers "Where's my order?" perfectly. But ask something complex li...
- 1 kudos
- 2163 Views
- 7 replies
- 5 kudos
Resolved! configure AWS authentication for serverless Spark
I only have an AWS Access Key ID and Secret Access Key, and I want to use this information to access S3.However, the official documentation states that I need to set the AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID environment variables, but I cannot ...
- 2163 Views
- 7 replies
- 5 kudos
- 5 kudos
Hi @liu ,The proper way is to go to your cluster and in advanced section you can set them up. In that way they will be scoped at cluster level. It's recommended to store values itself in a secret scopes as environment variables:Use a secret in a Spa...
- 5 kudos
- 706 Views
- 1 replies
- 1 kudos
Resolved! is there another way to authen to azure databricks using MSI on Java
Hi I am try to connect to azure databricks using MSI on Java but on a documenthttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/azure-miit saidThe Databricks SDK for Java has not yet implemented Azure managed identities authentication...
- 706 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @touchyvivace Unfortunately not, the documentation is up to date. In the Java SDK, MSI has not been implemented yet.And here's an open issue on github:[FEATURE] Add support for Azure Managed Identity authentication (system and user-assigned) · Iss...
- 1 kudos
- 1101 Views
- 1 replies
- 2 kudos
Dev/Pie/Prd and the same workspace
Hi all!I'm appealing to all you folk who are clever than I for some advice on Databricks dev ops.I was asked by my team leader to expand our singular environment to a devops style dev/pie/prd system, potentially using Dabs to promote code to higher e...
- 1101 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @IONA ,I guess you can still use DABs to simulate different environments on single workspace. In targets define 3 different environments but with the same for all of them (something similar to the picture below).Then your intuition is good - it's ...
- 2 kudos
- 1137 Views
- 1 replies
- 1 kudos
ServiceNow LakeFlow Connector - Using TABLE API only for tables and NOT views
The current Databricks ServiceNow Lakeflow connector relies on ServiceNow REST TABLE API to capture data. And for some reason, it is unable to list a user defined view as a data source to be configured, even though ServiceNow user defined views are a...
- 1137 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @ckanzabedian, have you checked out the documentation yet for the ServiceNow connector?https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/servicenow-limits The link above is about the limits. I can't see a mention about ...
- 1 kudos
- 3565 Views
- 1 replies
- 1 kudos
Resolved! How to Find DBU Consumption and Cost for a Serverless Job?
Hello community,I'm new to using Serverless compute for my Jobs and I need some help understanding how to monitor the costs.I have configured and run a job that executes a notebook using the "Serverless" compute option. The job completed successfully...
- 3565 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @LonguiVic1 ,You can use system table to track consumption in serverless. In below article they even provide sample queries you can use. Also, notice that there's list_prices system table that includes list prices over time for each available SKU....
- 1 kudos
- 966 Views
- 1 replies
- 0 kudos
Databricks asset bundles in Python: referencing variables
Hey,I am using DAB's and in my .yml files I can reference my variables set in my databricks.yml like this: git_branch: ${var.branch}I would like to do the same thing in my DAB's written in Python but I cannot find any documentation on how to do this....
- 966 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @daan_dw ,To reference variables defined in your databricks.yml in Python DAB code, define your variables class and use bundle.resolve_variablehttps://docs.databricks.com/aws/en/dev-tools/bundles/python/#access-bundle-variables
- 0 kudos
- 1596 Views
- 1 replies
- 1 kudos
Resolved! Online Table Migration
I am currently trying to migrate our Online Tables to synced tables with Online Feature Store since Online Tables is deprecated. When creating a new table, it worked just fine and how the docs said it would (https://docs.databricks.com/aws/en/machine...
- 1596 Views
- 1 replies
- 1 kudos
- 1 kudos
Migrating from deprecated Online Tables to synced tables with the Databricks Online Feature Store can be tricky due to several points of integration and timing between Unity Catalog (UC), Feature Store metadata, and the underlying online store. The m...
- 1 kudos
- 3097 Views
- 1 replies
- 0 kudos
Resolved! Spark JDBC Netsuite error - SQLSyntaxErrorException: [NetSuite][OpenAccess SDK JDBC Driver][OpenAcc
I'm trying to query the Customer netsuite tables with spark jdbc and I've added the .jar file to the cluster and trying to run the below:jdbc_url = "jdbc:ns://xxxx.connect.api.netsuite.com:1708;ServerDataSource=NetSuite2.com;Encrypted=1;NegotiateS...
- 3097 Views
- 1 replies
- 0 kudos
- 0 kudos
If you are on a Spark version that supports .option("query", ...), you can do: python df = spark.read \ .format("jdbc") \ .option("url", jdbc_url) \ .option("query", "SELECT TOP 10 * FROM Customer") \ .option("user", "xxxx") \ ...
- 0 kudos
- 795 Views
- 2 replies
- 1 kudos
Declarative Pipeline Failure for Autoloader
Hello Folks, After moving my working serverless Auto Loader notebook to a declarative (DLT) pipeline, I’m getting an AccessDenied error. What could be causing this?”here is the DLT json and error message in the DLT I googled around got saw some hint...
- 795 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello @iskidet! Were you able to resolve the AccessDenied issue? If the above suggestion helped, or if you found another solution, it would be great if you could mark it as the accepted solution or share your approach with the community.
- 1 kudos
- 2744 Views
- 7 replies
- 4 kudos
Resolved! [CANNOT_OPEN_SOCKET] Can not open socket: ["tried to connect to ('127.0.0.1', 45287)
Hello,after databricks update the Runtime from Release: 15.4.24 to Release: 15.4.25 we getting in all jobs the Error:[CANNOT_OPEN_SOCKET] Can not open socket: ["tried to connect to ('127.0.0.1', 45287)What we can do here?Greetings
- 2744 Views
- 7 replies
- 4 kudos
- 4 kudos
Hi @Hansjoerg,Apologies for the confusion earlier. You are right Bundles doesn't allow pinning to specific patch versions like 15.4.24.Your best option is to skip Bundles for now and use the regular Databricks Jobs setup (via UI or Jobs API) where yo...
- 4 kudos
- 2470 Views
- 5 replies
- 2 kudos
Read files from adls in databricks
I have unity catalogue access connector but its not enabled as i have only admin access so i dont have access to the admin portal to enable this as its need global admin permissions.I am trying to read adls json data in databricks by using service pr...
- 2470 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @SuMiT1 once networking issue is resolved , also make sure your service principal has at least Storage Blob Data Reader on the storage account/container.
- 2 kudos
- 3208 Views
- 3 replies
- 2 kudos
Resolved! Convert notebook to Python library
Looking for ways to convert a Databricks notebook to Python library. Some context :Don't want to give execute permissions to shared notebooks as we want to hide code from users.Proposed solution is to have our shared notebook converted into a Python ...
- 3208 Views
- 3 replies
- 2 kudos
- 2 kudos
The best way to share code from a Databricks notebook as a reusable module while hiding implementation details from users—without using wheels or granting direct notebook execution permissions—is to convert your notebook into a Python module, store i...
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 580 | |
| 349 | |
| 287 |
