Is there any way to create Databricks jobCluster through Databricks connect We are using All purposed cluster so far, to reduce Databricks cost we are planning to go ahead with jobCluster but unfortunately, I couldn't find a way to create jobCluster ...
I want to create an online chat bot that answers questions related to the immigration visa category that we work in. I don’t have programming experience myself. I would like to find someone in this community that could lead our product development.
I suppose you are looking to recruit someone? You might be lucky finding someone in here, but job boards might have a larger reach.
In ADF, a variable called Datetime was specified to be entered and executed in a workflow called Parent using Web Activity.There is a Child Workflow (two-step workflow) in Parent Workflow, and Datetime variables are available in Parent Workflow, but ...
the first attached image runs fine when cluster startsthis next image of the script does not finish when I include the -y parameter. if I include the -y I get failed: Script exit status is non-zero can someone please help
Hi, if I have understand your issue correctly, when you run directly in notebook it ends with an error but when you run it inside .sh with -y it runs okay?
I am using databricks-connect and VS Code to develop some python code for Databricks.I would like to code and run/test everything directly from VS Code using databricks-connect to avoid dealing with Databricks web IDE. For basic notebooks, it works j...
I am working on a migration project to migrate the HDFS command executions I do inside my Python code via os.system() function by replacing it with dbutils functions.dbutils functions are working as expected if I pass the fully qualified path of a fi...
As I mentioned in my problem statement, I have already implemented the required functionality with alternative approaches (AWS S3 API and BOTO3 API).Still, it is an outstanding issue with DBUTILS.
I am trying to deploy a model in the serving endpoints section, but it keeps failing after attempting to create for an hour. Here are the service logs:Container failed with: 9 +0000] [115] [INFO] Booting worker with pid: 115[2023-09-15 19:15:35 +0000...
Hello, How can I configure a foreign catalog connection to use SSH tunneling? I want to be able to use unity catalog
Hi, in addition to our previous message, you can try https://docs.databricks.com/en/query-federation/foreign-catalogs.html and https://grant-6562.medium.com/connecting-to-sql-server-through-an-ssh-tunnel-with-python-17de859caca5. Also please tag @Deb...
Hi there,I am having issue with writing a df to a table or display it. I have three dataframes that I have unioned and after I have done the union I cannot display the dataframe.df_table1 = spark.sql(f'SELECT * FROM {sql_full_name}')df_table2 = ...df...
Hi there,I am having issue with writing a df to a table or display it. I have three dataframes that I have unioned and after I have done the union I cannot display the dataframe.df_table1 = spark.sql(f'SELECT * FROM {sql_full_name}')df_table2 = ...df...
I am planning to add team names in custom tags and was hoping can do it with allowList and then have the user choose from the list. I am trying to avoid having multiple policy files per team.Has anybody found a good way to do this? May be using globa...
Hi, Currently, we have two different AWS accounts: dev and prod. We also have two different workspaces: one for dev and another for prod. The strange thing is that prod costs are being added to the dev account on AWS Cost Explorer ("Databricks Lakeho...
Databricks uses tags and AWS CloudTrail logs to connect and report costs to AWS.Tags can be used to monitor costs and attribute Databricks usage to different business units and teams.AWS CloudTrail logs can be used to calculate the exact cost of API ...
I am trying to fetch filter data based on date format on a date column. Below is the query forming in Databricks. In Databricks giving empty results.SELECT * FROM Test.TestSchema.Address where TO_DATE(TO_VARCHAR(MODIFIEDDATE,'YYYY-MM-DD')) = '2023-09...
I am trying to fetch filter data based on date format on a date column. Below is the query forming in Databricks. In Databricks giving empty results.SELECT * FROM Test.TestSchema.Address where TO_DATE(TO_VARCHAR(MODIFIEDDATE,'YYYY-MM-DD')) = '2023-09...
How do you set permission to just execute a query but cannot modify settings on the sql warehouse
Hi, Have you tried this: https://docs.databricks.com/en/security/auth-authz/access-control/sql-endpoint-acl.html Please let us know if this helps. Thanks!
Using SAS keys is a security issue that we would like to avoid, how do we utilize structured streaming from Event Hub while authenticating to Azure AD (client_id and secret).We know that we can use pythons Event Hub library, but that will make have t...
Hi, Could you please try structured streaming event hubs integration? https://docs.databricks.com/en/_extras/notebooks/source/structured-streaming-event-hubs-integration.html
Hi Team,Am new to databricks and please find my below question and do the favor.I have created a cluster then database, tables and data also inserted, Now I want to access this table data from my dotnet console application which is in 6.0 framework(v...
Hi, Databricks extension will help you to do so. https://learn.microsoft.com/en-us/azure/databricks/dev-tools/vscode-ext Please let us know if this helps. Thanks!
