- 1184 Views
- 1 replies
- 0 kudos
How to refer to repository directories in Workflows
HiI need to refer to a configuration file which resides in a separate directory than the script. The paths and execution withing a notebook/python files work fine. When script is scheduled and uses code repository the directory names look obfuscated ...
- 1184 Views
- 1 replies
- 0 kudos
- 0 kudos
can you try with relative path?like you get your current path within test_script.py with command like:cur_path = os.getcwd() then get the path to config.yaml with a relative path like:config_path = os.path.abspath(os.path.join(cur_path, f"../config/c...
- 0 kudos
- 969 Views
- 6 replies
- 0 kudos
Performance issue when reading data from a view.
Hi All,We are facing some performance issue and I need your help to know what could be the best approach to follow here.Existing: For each region, we have view(Reg1_View,Reg2_View..) to pull data from table( we don't have direct access to table).And ...
- 969 Views
- 6 replies
- 0 kudos
- 0 kudos
Does any table hold data of all region 1. if yes. Get a Materialized view created (replacing all_reg_view)2. i see you already tried creating a staging table replacing the all_reg_view. Try creating cluster key along with partition.Cluster key on the...
- 0 kudos
- 1725 Views
- 1 replies
- 0 kudos
No access to databricks console
Hello, I have the following situation: when trying to configure SSO, it was enabled to allow login using Microsoft, but the problem is that the sessions expired, and now we cannot access with any email, it says that the account is not enabled.How co...
- 1725 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @thiagoawstest, Please reach out to your Account Executive or Solutions Architect. The will be able to help you with the issue you're experiencing while trying to log in. Best
- 0 kudos
- 1711 Views
- 1 replies
- 0 kudos
Failed to send request to Azure Databricks Cluster
We have scheduled an ADF (Azure Data Factory) pipeline that contains a Lookup activity, which is responsible for fetching the last ingested date from the Databricks catalog(Hive metastore). I attached screenshot please find https://yourimageshare.com...
- 1711 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @databricks98, It seems like there is some issue connecting to your Azure account. Were there any recent changes to firewalls, permissions, or cluster configurations? Could you please check to make sure that the connection between Databricks and ...
- 0 kudos
- 53800 Views
- 25 replies
- 16 kudos
How to import local python file in notebook?
for example I have one.py and two.py in databricks and I want to use one of the module from one.py in two.py. Usually I do this in my local machine by import statement like below two.py__ from one import module1 . . . How to do this in databricks???...
- 53800 Views
- 25 replies
- 16 kudos
- 16 kudos
This alternative worked for us: https://community.databricks.com/t5/data-engineering/is-it-possible-to-import-functions-from-a-module-in-workspace/td-p/5199
- 16 kudos
- 433 Views
- 2 replies
- 0 kudos
Resolved! How can we fetch application id in serverless compute in databricks?
Hi All,How can we fetch application id in serverless compute in databricks? We are working to use serverless compute for some jobs. The issue is we are not able to fetch application id in notebook. Earlier we were using spark.sparkContext.application...
- 433 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @momita_s, Thanks for your question. As you mentioned above parameters/commands are not available in serverless, I did not find a way to retrieve the applicationID on serverless, I would check internally but this likely requires a feature request.
- 0 kudos
- 579 Views
- 5 replies
- 0 kudos
Unable to find course materials for the course: Delivery Specialization: CDW Migration Best Practice
Hi,Tried searching everywhere but unable to find course materails for the course: Delivery Specialization: CDW Migration Best Practice.In the video it says see course materials, but I dont see anything with that name anywhere. Please let me know if s...
- 579 Views
- 5 replies
- 0 kudos
- 0 kudos
I believe it is closed, I'm unable to check the status of the ticket. It is giving an error: "You currently do not have access to Help Center.Please reach out to your admin or send an email to help@databricks.com"If you are able to check the request ...
- 0 kudos
- 667 Views
- 5 replies
- 0 kudos
Apply-changes-table (SCD2) with huge amounts of `rowIsHidden=True` rows
I have a lot of DLT tables creating using the `apply_changes` function with type 2 history. This functions creates a physical table `__apply_changes_storage_<table_name>` and a view on top of this `<table_name>`. The number of rows the physical table...
- 667 Views
- 5 replies
- 0 kudos
- 0 kudos
I'm trying, but doesn't seem to change anything. Setting these table properties - when are the "applied"? When the job is run, or as a background thing?
- 0 kudos
- 538 Views
- 1 replies
- 1 kudos
Resolved! Delta Live Tables - use cluster-scoped init scripts
Hi All,According the documentation of delta live tables (https://docs.databricks.com/en/delta-live-tables/external-dependencies.html), one can user either global or cluster scoped init scripts. However, I don't see an option to select init scripts in...
- 538 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi, You can do it via a cluster policy. It can be achieved in two steps: create a cluster policy with required attributes.You can provide the init_scripts attribute in the policy. For reference: https://learn.microsoft.com/en-us/azure/databricks/admi...
- 1 kudos
- 286 Views
- 1 replies
- 0 kudos
Link for webinar Get Started with Databricks for Data Engineering session
where to find Link for webinar Get Started with Databricks for Data Engineering session
- 286 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Meenambigai! If you have successfully enrolled in the course, open the Databricks Academy, click on the kebab menu icon (upper left corner), select "My Calendar". You’ll see the courses you’re enrolled in, organized by date. Click on the link ...
- 0 kudos
- 443 Views
- 2 replies
- 0 kudos
Resolved! Custom library in clean rooms
Hello Hello ! I want to use a clean room to run some algorithms developed for one of my customer without exchanging any data, the code is stored as a python library in a private git repo connected to databricks. 1 - We'd like to import the library in...
- 443 Views
- 2 replies
- 0 kudos
- 587 Views
- 3 replies
- 0 kudos
How to add shared libs
Hi team,I want to add some shared libs which might be used by many repos, e.g. some util functions which might be used by any repos.1. What is the recommended way to add those libs? E.g. create a separate repo and reference it in another repo?2. How ...
- 587 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Brad Typically, You specify shared libraries in init script. From there, init script will be executed for each job compute, ensuring lib consistency.The other way - You could use a job cluster policy and specify desired libraries that will be pro...
- 0 kudos
- 1508 Views
- 4 replies
- 2 kudos
Serverless compute databricks
1. How to connect s3 bucket to databricks since dbfs mount is not supported.?2. In serverless compute Spark Context (sc), spark.sparkContext, and sqlContext are not supported?. Does it means it will not leverage power of distributed processing?3. Wha...
- 1508 Views
- 4 replies
- 2 kudos
- 2 kudos
please see this documentation for accessing cloud storage by setting Unity Catalog objects: Storage Credential and External Location. https://docs.databricks.com/en/connect/unity-catalog/cloud-storage/index.html
- 2 kudos
- 10409 Views
- 3 replies
- 2 kudos
Is ABAC feature enabled?
Can anyone please share me the example of how it works in terms of access controls?
- 10409 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Naveena G​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers you...
- 2 kudos
- 902 Views
- 2 replies
- 2 kudos
Code coverage on Databricks notebook
I have a scenario where my application code a scala package and notebook code[Scala] under /resources folder is being maitained.I am trying to look for a easiest way to perform code coverage on my notebook , does Databricks provide any option for it....
- 902 Views
- 2 replies
- 2 kudos
- 2 kudos
Important thing which missed to add in post is , we do maintan notebook code as .scala under resources and maitian in github. Files(.scala) from resources gets deployed as notebook using github action.With my approach of moving under package, I will ...
- 2 kudos
| User | Count |
|---|---|
| 1613 | |
| 770 | |
| 349 | |
| 286 | |
| 253 |
